A party attacks a creature with a lot of hit points and someone casts Sleep. The creature resists the spell because it has more hit points than what was rolled by the caster for the sleep spell, but can the sleep effect be triggered later once the creature loses enough hit points to fall below that sleep threshold?

For example: The party fights an enemy creature with 30 hit points. The wizard casts Sleep, the enemy is the only creature in the area of effect, and rolls 5d8 for a total of 20. The creature resists because it has more than 20 hit points. After a couple attacks, the creature is brought down to 18 hit points and we are still within the 1 minute duration of the Sleep spell. Would the sleep effect then get activated since the creature now has less than 20 hit points?

As context for the question, I saw this scenario play out in a season 1 episode of the Critical Role webcast where a Fomorian first resisted a Sleep spell, only to fall asleep after suffering enough damage, even though the Sleep spell was cast a few turns earlier. I know that the DM Matt Mercer plays a modified version of the 5E rules, especially since season 1 was converted from Pathfinder, but since I've only recently started playing 5E, I actually thought that's how the spell worked until a Wizard in my group cast it last night and we spent time re-reading the spell details.

After further research and discussion, it turns out a second sleep arrow was shot by Vex.^[1][2][3] So while it did not happen this way in Critical Role, the question still stands on its own since I genuinely thought at the time that it was how the Sleep spell worked in 5E.

No

The sleep spell says:

Creatures within 20 feet of a point you choose within range are affected in ascending order of their current hit points [...] A creature's hit points must be equal to or less than the remaining total for that creature to be affected.

If a creature's hit points are not equal to or less than the remaining total, that creature is not 'affected', and the duration doesn't matter to them.

DMs often play loose with the rules, as in the case of Critical Role, usually to hurry along a fight or end it in a particular way even though that particular outcome is not actually possible by the rules as written.

No, it shouldn't. The sleep effect is only initiated at the beginning of the spell, the duration is how long they are kept asleep for (assuming they take no damage or aren't woken up).

Otherwise, given your example, each of your buddies would fall asleep as they walked into the AoE and used up the HP worth of sleep spell. As a side note, when cast, it also should have affected any of your buddies in range, as sleep does not differentiate between friend and foe. It starts with the creature with the lowest HP, and goes up from there, in a 20ft radius from center point.

No, Sleep doesn't have an effect in later rounds

By RAW No, as other answers have correctly stated this is not how the sleep spell works and is not how to should be ruled. Even if they would fall asleep at later turns they would sleep for less time as the 1 minute is counted from casting time.

I thought this was a cool idea and made a homebrew spell that does work the way you describe. I am including it below with an explanation of the differences to show why the RAW sleep spell doesn't work this way.

What sleep would look like if it did work this way

Here's how I would create a homebrew version of sleep that does function like this. It doesn't quite fit the way spells usually work in DND 5e but then again neither does RAW Sleep.

Higher Spell Level

Sleep is a first level spell that is a available to a bunch of classes. This change absolutely increases the power of the spell and will need to be cast in a higher slot. I'm going to suggest 3rd level for now as that is about the right level if we consider the 5d8 to be straight spell damage.

Have a saving throw

Without a save this is a very strong spell, It would be a Constitution or Wisdom save depends on how you want to flavour the spell. Con for a physical effect and Wis for a mental one.

Longer duration

If you want this to be worth casting early in a fight then you need it to last until the end. Yes most fights last less than a minutes but I imagine you might like to cast it before combat even starts. I would therefore increase the duration to 10 minutes.

With those changes taken into consideration here is my homebrew modified sleep spell that works as you describe.

Deep Sleep

3rd level enchantment

Casting Time: 1 action

Range: 90 feet

Components: V S M (A handful of sand, rose petals or a cricket)

Duration: 10 minutes

Save: Wisdom, negates

Classes: Bard, Sorcerer, Wizard

This spell causes creatures to feel magically sleepy. Choose a point you can see within range, creatures within 20 feet of this point must make a Wisdom Saving Throw. On a failed save the creature feels sleepy, roll 5d8 if at any point for the duration the creatures current hit points fall below this total they fall asleep.
Multiple castings of this spell do not stack.

At Higher Levels: When you cast this spell using a spell slot of 2nd level or higher, roll an additional 2d8 for each slot level about 3rd.

There is actually precedent for the way this spell works, just not in 5e. This is very similar to the way non-lethal damage is handled in Pathfinder. Effectively it raises the 0 point for going unconscious rather than reducing current hit points.

You should be able to see by how strong this improved version is as to why RAW Sleep doesn't and shouldn't work this way.