Float precision problem in metal shading language
up vote
0
down vote
favorite
I pass a float param with value 0.00291545 to fragment shader.
But fragment function got n/a .
swift code
fragment shader
In Xcode debugger's "Bound Resources" view, it shows that the value is 0.003
Bound Resources
In Xcode shader debugger view, I made some testes. It shows that 0.0004 will be casted to 0 and 0.0005 will be casted to 0.001 . The float value behaves like half value.
test code in shader debugger
So, my question is:
Why 0.00291545 is first casted to 0.003, then casted to 0 ?
Why float value has the same precision with half ?
How could I use float value that has normal precision ?
ios swift precision metal shading
asked Nov 10 at 17:57
blabla
33
add a comment |
up vote
0
down vote
favorite
I pass a float param with value 0.00291545 to fragment shader.
But fragment function got n/a .
swift code
fragment shader
In Xcode debugger's "Bound Resources" view, it shows that the value is 0.003
Bound Resources
In Xcode shader debugger view, I made some testes. It shows that 0.0004 will be casted to 0 and 0.0005 will be casted to 0.001 . The float value behaves like half value.
test code in shader debugger
So, my question is:
Why 0.00291545 is first casted to 0.003, then casted to 0 ?
Why float value has the same precision with half ?
How could I use float value that has normal precision ?
ios swift precision metal shading
asked Nov 10 at 17:57
blabla
33
Please post textual information (like source code) as text, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That does not mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise.
– Ken Thomases
Nov 10 at 20:07
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I pass a float param with value 0.00291545 to fragment shader.
But fragment function got n/a .
swift code
fragment shader
In Xcode debugger's "Bound Resources" view, it shows that the value is 0.003
Bound Resources
In Xcode shader debugger view, I made some testes. It shows that 0.0004 will be casted to 0 and 0.0005 will be casted to 0.001 . The float value behaves like half value.
test code in shader debugger
So, my question is:
Why 0.00291545 is first casted to 0.003, then casted to 0 ?
Why float value has the same precision with half ?
How could I use float value that has normal precision ?
ios swift precision metal shading
asked Nov 10 at 17:57
blabla
33
I pass a float param with value 0.00291545 to fragment shader.
But fragment function got n/a .
swift code
fragment shader
In Xcode debugger's "Bound Resources" view, it shows that the value is 0.003
Bound Resources
In Xcode shader debugger view, I made some testes. It shows that 0.0004 will be casted to 0 and 0.0005 will be casted to 0.001 . The float value behaves like half value.
test code in shader debugger
So, my question is:
Why 0.00291545 is first casted to 0.003, then casted to 0 ?
Why float value has the same precision with half ?
How could I use float value that has normal precision ?
ios swift precision metal shading
ios swift precision metal shading
asked Nov 10 at 17:57
blabla
33
asked Nov 10 at 17:57
blabla
33
asked Nov 10 at 17:57
blabla
33
asked Nov 10 at 17:57
blabla
33
asked Nov 10 at 17:57
blabla
33
33
Please post textual information (like source code) as text, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That does not mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise.
– Ken Thomases
Nov 10 at 20:07
add a comment |
Please post textual information (like source code) as text, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That does not mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise.
– Ken Thomases
Nov 10 at 20:07
Please post textual information (like source code) as text, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That does not mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise.
– Ken Thomases
Nov 10 at 20:07
Please post textual information (like source code) as text, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That does not mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise.
– Ken Thomases
Nov 10 at 20:07
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
I found what the problem was!
Xcode debugger misled me.
0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate， but the value is accurate actually.
Thanks for Ken Thomases
answered Nov 11 at 11:42
blabla
33
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
I found what the problem was!
Xcode debugger misled me.
0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate， but the value is accurate actually.
Thanks for Ken Thomases
answered Nov 11 at 11:42
blabla
33
add a comment |
up vote
0
down vote
I found what the problem was!
Xcode debugger misled me.
0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate， but the value is accurate actually.
Thanks for Ken Thomases
answered Nov 11 at 11:42
blabla
33
add a comment |
up vote
0
down vote
up vote
0
down vote
I found what the problem was!
Xcode debugger misled me.
0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate， but the value is accurate actually.
Thanks for Ken Thomases
answered Nov 11 at 11:42
blabla
33
I found what the problem was!
Xcode debugger misled me.
0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate， but the value is accurate actually.
Thanks for Ken Thomases
answered Nov 11 at 11:42
blabla
33
answered Nov 11 at 11:42
blabla
33
answered Nov 11 at 11:42
blabla
33
answered Nov 11 at 11:42
blabla
33
33
add a comment |
add a comment |
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53241847%2ffloat-precision-problem-in-metal-shading-language%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Please post textual information (like source code) as text, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That does not mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise.
– Ken Thomases
Nov 10 at 20:07