This paper consists of musings that originate mainly from conversations with other physicists, as together we've tried to learn some cryptography, but also from conversations with a couple of classical cryptographers. The main thrust of the paper is an attempt to explore the ramifications for cryptographic security of incorporating physics into our thinking at every level. I begin by discussing two fundamental cryptographic principles, namely that security must not rely on secrecy of the protocol and that our local environment must be secure, from a physical perspective. I go on to explain why by definition a particular cryptographic task, oblivious transfer, is inconsistent with a belief in the validity of quantum mechanics. More precisely, oblivious transfer defines states and operations that do not exist in any (complex) Hilbert space. I go on to argue the fallaciousness of a "black box" approach to quantum cryptography, in which classical cryptographers just trust physicists to provide them with secure quantum cryptographic sub-protocols, which they then attempt to incorporate into larger cryptographic systems. Lest quantum cryptographers begin to feel too smug, I discuss the fact that current implementations of quantum key distribution are only technologically secure, and not "unconditionally" secure as is sometimes claimed. I next examine the concept of a secure lab from a physical perspective, and end by making some observations about the cryptographic significance of the (often overlooked) necessity for parties who wish to communicate having established physical reference frames.