When I sign a message/transaction¹ with a private key,
and I get a message/transaction¹ signature,
Critically, it's used different public/private key pairs in these two things. Nothing in the rest of the question is about the first of these two things. Everything is about verifying the signature in the second of these things, against the public key of the public/private key pair used by the signer in said second thing. Said public key is assumed available to the verifier. The matching private key is not.
With private/public key pairs, the point of the technology is that you cannot reverse engineer the private key from the public key
Correct. There's slightly more to it: with the public key it's not possible to perform the things that the private key is intended to perform.
What is the difference between reverse engineering a private key from a public key, and verifying that someone must have the private key to produce a message signature that corresponds to a public key? How is one possible but not the other?
Signature works according to this diagram:
and by a modern textbook
- $1^n$ encodes integer $n$ defining the size of keys. In practice $n$ is fixed and public.
- $(\mathrm{pk},\mathrm{sk})$ is a public/private key pair output by the key generation algorithm $\mathrm{Gen}$. It's assumed $\mathrm{sk}$ is kept secret by the assigned owner of the key pair, who most often is the one who ran $\mathrm{Gen}$.
- $m$ is the message to sign (an arbitrary bitstring, save perhaps for size requirements).
- $\sigma$ is the signature of the message. It's produced from $\mathrm{sk}$ and $m$ by the signature algorithm $\mathrm{Sign}$.
- $b$ is the integrity indicator, which takes one of two values, Valid or Invalid. It's produced from $\mathrm{pk}$, $m$ and $\sigma$ by the verification algorithm $\mathrm{Vrfy}$.
A signature scheme is correct when, with things as per the drawing, $b$ always is Valid. It's secure² when adversaries given $\mathrm{pk}$ and the ability to obtain $\sigma_i$ for any $m_i$ they see fit, are unable to exhibit an $(m,\sigma)$ pair with $\mathrm{Vrfy}(\mathrm{pk},m,\sigma)$Valid, and $m\ne m_i$ for any $i$. There are a few more technical details³.
It's surprising that there are correct and secure signature schemes. Devising one was long in the making. But it's no more surprising than the possibility of public key encryption. If one mostly cares for understanding the use of signature, an option is to admit there are such signature schemes.
Another option is to study one. I suggest Schnorr signature (alt. version), which principle is used in some cryptocurrencies, is perhaps the simplest, and has a relatively simple quantitative security reduction to the hardness of the Discrete Logarithm Problem in the group used, under a random oracle model of it's hash. Exposing it would about triple the length of the answer.
¹ In a cryptocurrency context, a message may describe a transaction.
² By the Existentially UnForgeable under Chosen Message Attack criteria, often the only one discussed in modern introductory exposition. There are other useful signature security criteria.
³$\mathrm{Gen}$, $\mathrm{Sign}$, $\mathrm{Vrfy}$ and adversaries are modeled as probabilistic polynomial time algorithms. Propositions are stated for any fixed $(\mathrm{pk},\mathrm{sk})$ pair output by $\mathrm{Gen}$, and except with negligible probability $p(n)$, that is $p(n)$ such that for any polynomial $Q(n)$ it holds $\displaystyle 0=\lim_{n\to\infty} p(n)\,Q(n)$. In practice $n$ is chosen high enough to make that probability practically negligible.