In a word: no. Yes, your readers will need to download the book, but if you're selling your book on Amazon, they click the link and your book is downloaded in the background. The next time they open their ereader, the book should be there. But is there more to it than that? Find out if there are file sizes recommended by Amazon, limits to internal images or the cover image, and whether or not worrying about the file size is important.
Read the full article: File Sizes for Kindle Books