I would translate Deuteronomy 16:1-3 like this:
1 Mark the month of Abib, and prepare the Passover to the LORD your God, since in the month of Abib the LORD your God brought you forth from Egypt by night.
2 And you shall sacrifice the Passover to the LORD your God from the flock and herd in the place where it is acceptable to the LORD to place his name.
3 You will not eat leaven with it. Seven days you will eat unleavened bread with it ‒ the bread of affliction [you ate] when you came forth in haste from the land of Egypt. This is how you will remember the day of your coming forth from the land of Egypt, all the days of your life.
Details:
"Unleavened bread" is called "flatbread" in many cultures, and principally consists of two ingredients: flour and water, which is why it is also referred to as "poor man's bread". It can be made and then immediately baked, since the absence of leaven means there is no need for the resting and prolonged kneading that leavened bread requires.
According to Exodus 12, some time prior to Abib 10 (1) God informed Moses and Aaron that he intended to bring the Hebrews out of Egypt on the night of the fourteenth (beginning after sunset of the thirteenth). They were to eat the Passover "in haste" (Exodus 12:11), and were urged by the Egyptians to go "in haste" from their land (Exodus 12:33), so they took with them only the unleavened dough they had in their kneading bowls (Exodus 12:34).
The hasty departure from Egypt meant that the Hebrews would have had limited supplies, and it was not until some time after Ziv 15 that God provided quail and manna for them (Exodus 16:13-14) -- in response to the hunger which caused them to look back with fondness on life in Egypt (Exodus 16:2-3).
The noun עֳנִי (Strong's H6040 - `oniy) means "affliction/poverty/misery", and is related to the adjective עָנִי (Strong's H6041 - `aniy), which means "poor/afflicted/humble/wretched". Both words being derived from the verb עָנָה (Strong's H6031 - `anah).
The Hebrews left Egypt in possession of significant wealth (Exodus 12:34-35), but it would have been of no use in procuring food for themselves in the wilderness. So, food-wise, they where indeed paupers.
Regardless of whether לֶ֣חֶם עֹ֑נִי is translated "poor man's bread" or "bread of affliction" the intended meaning relates to poverty, since the affliction of the poor man is the paucity (quality and quantity) of his daily bread.
Notes:
(1) God said to Moses and Aaron, "This month shall be your beginning of months" (Exodus 12:2). So, they were in the month of Abib. The requirement concerning the lamb (Exodus 12:3) indicates that Abib 10 had not yet arrived.