Comments on: A/B testing 2 subject lines http://blog.mailchimp.com/ab-testing-2-subject-lines/ MailChimp, email marketing, and monkeys! Thu, 09 Feb 2012 17:36:42 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Benhttp://blog.mailchimp.com/ab-testing-2-subject-lines/#comment-7995 Ben Sat, 08 May 2010 13:54:39 +0000 http://blog.mailchimp.com/?p=3038#comment-7995 A/B testing is not available on RSS campaigns. When we built RSS-to-email, it was designed for bloggers who just want another (automated) way to stay in touch with their followers. It's extremely powerful and useful, but was always seen as a "supplement" to your "normal" email marketing activities. So you'll notice A/B testing, TimeWarp, etc., is not going to be in the RSS area. A/B testing is not available on RSS campaigns. When we built RSS-to-email, it was designed for bloggers who just want another (automated) way to stay in touch with their followers. It’s extremely powerful and useful, but was always seen as a “supplement” to your “normal” email marketing activities. So you’ll notice A/B testing, TimeWarp, etc., is not going to be in the RSS area.

]]> By: Michael Hayeshttp://blog.mailchimp.com/ab-testing-2-subject-lines/#comment-7989 Michael Hayes Fri, 07 May 2010 21:14:03 +0000 http://blog.mailchimp.com/?p=3038#comment-7989 I'd also like to be able to run an A/B test on RSS-generated campaigns (knowing what time of day is best would also be useful).Is it possible? I’d also like to be able to run an A/B test on RSS-generated campaigns (knowing what time of day is best would also be useful).

Is it possible?

]]> By: Emma McCrearyhttp://blog.mailchimp.com/ab-testing-2-subject-lines/#comment-6532 Emma McCreary Wed, 06 Jan 2010 04:14:38 +0000 http://blog.mailchimp.com/?p=3038#comment-6532 pretty cool - is it possible to A/B test RSS-generated campaigns? It would be cool to be able to test different subject line formats. pretty cool – is it possible to A/B test RSS-generated campaigns? It would be cool to be able to test different subject line formats.

]]> By: Benhttp://blog.mailchimp.com/ab-testing-2-subject-lines/#comment-3561 Ben Mon, 06 Apr 2009 20:10:40 +0000 http://blog.mailchimp.com/?p=3038#comment-3561 Hi Lary, here are my thoughts:* It's possible a subject line was so spammy, it got spam filtered or bounced back (some spam filters send deceptive bouncebacks to make you clean your list)* I think that no matter what you're testing, it's valuable to see all data. You may discover a horrible subject line generated too many unsubs. Or a delivery time got a lot of bouncebacks from busy ISPs.* In terms of the 1.6% difference, yes and no. The difference might've been more dramatic if I waited longer than 6 hrs, but like I said, I'm too impatient. 1.6% sounds like a close call, but it can be huge if your list is large (a difference of hundreds of clicks for some of our users). If you're an e-retailer, you need every click-to-conversion you can get.Still, if you want complete control, the "manual" option is definitely the way to go. Hi Lary, here are my thoughts:

* It’s possible a subject line was so spammy, it got spam filtered or bounced back (some spam filters send deceptive bouncebacks to make you clean your list)

* I think that no matter what you’re testing, it’s valuable to see all data. You may discover a horrible subject line generated too many unsubs. Or a delivery time got a lot of bouncebacks from busy ISPs.

* In terms of the 1.6% difference, yes and no. The difference might’ve been more dramatic if I waited longer than 6 hrs, but like I said, I’m too impatient. 1.6% sounds like a close call, but it can be huge if your list is large (a difference of hundreds of clicks for some of our users). If you’re an e-retailer, you need every click-to-conversion you can get.

Still, if you want complete control, the “manual” option is definitely the way to go.

]]> By: Lary Stuckerhttp://blog.mailchimp.com/ab-testing-2-subject-lines/#comment-3560 Lary Stucker Mon, 06 Apr 2009 19:05:57 +0000 http://blog.mailchimp.com/?p=3038#comment-3560 Thanks for the nice example of how easy it is to create a/b testing with MailChimp.A couple of thoughts questions:- Other than establish a baseline for your other statistics how does bounce rates have a "winner" and a "losers" in a/b testing.- If you are just testing the subject line isn't the only metrics worth considering open rates and click-throughs?- With a 1.6% difference in open rates and a dead even click-through rate isn't the results to statistically close to call a winner? Thanks for the nice example of how easy it is to create a/b testing with MailChimp.

A couple of thoughts questions:

- Other than establish a baseline for your other statistics how does bounce rates have a “winner” and a “losers” in a/b testing.

- If you are just testing the subject line isn’t the only metrics worth considering open rates and click-throughs?

- With a 1.6% difference in open rates and a dead even click-through rate isn’t the results to statistically close to call a winner?

]]>