FIltering out peaks and valleys in a noisy dataset that has drift and hills
$begingroup$
I have a dataset from a weight sensor over 14 hours, whos resitance wheter or not something is on a table, which reads one value every second and stores it in a database.
In the end the senosor is supposed to play a beep when something is placed and add an entry in the database.
I have marked the valleys with red anthat detectsd the peaks in green.
Notice how all the valleys logarithmicly drift down, this is because of the inherent drift of weight sensors.
The first peak A is much higher than B, the reduction of restiance takes much longer the more time goes on.
So valley Z is higher than peak F.
Also notice the hills in valley V, which don't count as peaks since they are too small, and the strange noise next to E, which managed to reach higher than E.
Now my problem is I need to continuously detect these valleys as the values are comming in, igoring peaks smaller than 1 second, since they are most likely noise.
The problem I'm having is the drift, which prevents me from setting a threshold and the descending valleys in Z and Y which seam to almost gradually form compared to the transition between V and D which is almost instant.
Basicly I need a formula or a programm that can detect valleys as they are comming in, that accounts for mini hills and the drift aswell as the random noise.
I thought about using the average or the median the last 6 values and comparing it to the median of the next 2 values (to keep the delay under 2 seconds) and then playing the beep once the difference is over 4%.
The problem here are the gradual valleys that happen at the start, which I have no idea how to account for.
Also the percentile difference between peaks and valleys shrinks as time goes on.
Anybody have any idea how to reliably differentiate between peaks and valleys?
statistics python
asked Jan 22 at 8:55
user2741831user2741831
1466
$endgroup$
add a comment |
$begingroup$
I have a dataset from a weight sensor over 14 hours, whos resitance wheter or not something is on a table, which reads one value every second and stores it in a database.
In the end the senosor is supposed to play a beep when something is placed and add an entry in the database.
I have marked the valleys with red anthat detectsd the peaks in green.
Notice how all the valleys logarithmicly drift down, this is because of the inherent drift of weight sensors.
The first peak A is much higher than B, the reduction of restiance takes much longer the more time goes on.
So valley Z is higher than peak F.
Also notice the hills in valley V, which don't count as peaks since they are too small, and the strange noise next to E, which managed to reach higher than E.
Now my problem is I need to continuously detect these valleys as the values are comming in, igoring peaks smaller than 1 second, since they are most likely noise.
The problem I'm having is the drift, which prevents me from setting a threshold and the descending valleys in Z and Y which seam to almost gradually form compared to the transition between V and D which is almost instant.
Basicly I need a formula or a programm that can detect valleys as they are comming in, that accounts for mini hills and the drift aswell as the random noise.
I thought about using the average or the median the last 6 values and comparing it to the median of the next 2 values (to keep the delay under 2 seconds) and then playing the beep once the difference is over 4%.
The problem here are the gradual valleys that happen at the start, which I have no idea how to account for.
Also the percentile difference between peaks and valleys shrinks as time goes on.
Anybody have any idea how to reliably differentiate between peaks and valleys?
statistics python
asked Jan 22 at 8:55
user2741831user2741831
1466
$endgroup$
add a comment |
$begingroup$
I have a dataset from a weight sensor over 14 hours, whos resitance wheter or not something is on a table, which reads one value every second and stores it in a database.
In the end the senosor is supposed to play a beep when something is placed and add an entry in the database.
I have marked the valleys with red anthat detectsd the peaks in green.
Notice how all the valleys logarithmicly drift down, this is because of the inherent drift of weight sensors.
The first peak A is much higher than B, the reduction of restiance takes much longer the more time goes on.
So valley Z is higher than peak F.
Also notice the hills in valley V, which don't count as peaks since they are too small, and the strange noise next to E, which managed to reach higher than E.
Now my problem is I need to continuously detect these valleys as the values are comming in, igoring peaks smaller than 1 second, since they are most likely noise.
The problem I'm having is the drift, which prevents me from setting a threshold and the descending valleys in Z and Y which seam to almost gradually form compared to the transition between V and D which is almost instant.
Basicly I need a formula or a programm that can detect valleys as they are comming in, that accounts for mini hills and the drift aswell as the random noise.
I thought about using the average or the median the last 6 values and comparing it to the median of the next 2 values (to keep the delay under 2 seconds) and then playing the beep once the difference is over 4%.
The problem here are the gradual valleys that happen at the start, which I have no idea how to account for.
Also the percentile difference between peaks and valleys shrinks as time goes on.
Anybody have any idea how to reliably differentiate between peaks and valleys?
statistics python
asked Jan 22 at 8:55
user2741831user2741831
1466
$endgroup$
I have a dataset from a weight sensor over 14 hours, whos resitance wheter or not something is on a table, which reads one value every second and stores it in a database.
In the end the senosor is supposed to play a beep when something is placed and add an entry in the database.
I have marked the valleys with red anthat detectsd the peaks in green.
Notice how all the valleys logarithmicly drift down, this is because of the inherent drift of weight sensors.
The first peak A is much higher than B, the reduction of restiance takes much longer the more time goes on.
So valley Z is higher than peak F.
Also notice the hills in valley V, which don't count as peaks since they are too small, and the strange noise next to E, which managed to reach higher than E.
Now my problem is I need to continuously detect these valleys as the values are comming in, igoring peaks smaller than 1 second, since they are most likely noise.
The problem I'm having is the drift, which prevents me from setting a threshold and the descending valleys in Z and Y which seam to almost gradually form compared to the transition between V and D which is almost instant.
Basicly I need a formula or a programm that can detect valleys as they are comming in, that accounts for mini hills and the drift aswell as the random noise.
I thought about using the average or the median the last 6 values and comparing it to the median of the next 2 values (to keep the delay under 2 seconds) and then playing the beep once the difference is over 4%.
The problem here are the gradual valleys that happen at the start, which I have no idea how to account for.
Also the percentile difference between peaks and valleys shrinks as time goes on.
Anybody have any idea how to reliably differentiate between peaks and valleys?
statistics python
statistics python
asked Jan 22 at 8:55
user2741831user2741831
1466
asked Jan 22 at 8:55
user2741831user2741831
1466
asked Jan 22 at 8:55
user2741831user2741831
1466
asked Jan 22 at 8:55
user2741831user2741831
1466
asked Jan 22 at 8:55
user2741831user2741831
1466
1466
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3082900%2ffiltering-out-peaks-and-valleys-in-a-noisy-dataset-that-has-drift-and-hills%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3082900%2ffiltering-out-peaks-and-valleys-in-a-noisy-dataset-that-has-drift-and-hills%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
