How to get Kafka timestamp for a record in Flink?










1















I have a Flink application that uses the FlinkKafkaConsumer.



I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



How can the timestamp be obtained?






java scala apache-kafka apache-flink kafka-consumer-api







share|improve this question




























    1















    I have a Flink application that uses the FlinkKafkaConsumer.



    I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



    How can the timestamp be obtained?






    java scala apache-kafka apache-flink kafka-consumer-api







    share|improve this question


























      1












      1








      1








      I have a Flink application that uses the FlinkKafkaConsumer.



      I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



      How can the timestamp be obtained?






      java scala apache-kafka apache-flink kafka-consumer-api







      share|improve this question
















      I have a Flink application that uses the FlinkKafkaConsumer.



      I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



      How can the timestamp be obtained?






      java scala apache-kafka apache-flink kafka-consumer-api




      java scala apache-kafka apache-flink kafka-consumer-api






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 16 '18 at 5:57







      codeshark

















      asked Nov 15 '18 at 8:41









      codesharkcodeshark

      1,93673875




      1,93673875






















          1 Answer
          1






          active

          oldest

          votes


















          1














          You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
          The timestamp type is indicated in timestampType().






          share|improve this answer






















            Your Answer






            StackExchange.ifUsing("editor", function ()
            StackExchange.using("externalEditor", function ()
            StackExchange.using("snippets", function ()
            StackExchange.snippets.init();
            );
            );
            , "code-snippets");

            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "1"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













            draft saved

            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53315359%2fhow-to-get-kafka-timestamp-for-a-record-in-flink%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1














            You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
            The timestamp type is indicated in timestampType().






            share|improve this answer



























              1














              You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
              The timestamp type is indicated in timestampType().






              share|improve this answer

























                1












                1








                1







                You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
                The timestamp type is indicated in timestampType().






                share|improve this answer













                You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
                The timestamp type is indicated in timestampType().







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 15 '18 at 10:04









                Axel PodehlAxel Podehl

                1,6861515




                1,6861515





























                    draft saved

                    draft discarded
















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …


                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.

                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53315359%2fhow-to-get-kafka-timestamp-for-a-record-in-flink%23new-answer', 'question_page');

                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    這個網誌中的熱門文章

                    Barbados

                    圖片
                    For other uses, see Barbados (disambiguation). Not to be confused with Barbuda. Coordinates: 13°10′N 59°33′W  /  13.167°N 59.550°W  / 13.167; -59.550 Barbados Flag Coat of arms Motto:  "Pride and Industry" Anthem:  In Plenty and In Time of Need Capital and largest city Bridgetown 13°06′N 59°37′W  /  13.100°N 59.617°W  / 13.100; -59.617 Official languages English Recognised regional languages Bajan Creole Ethnic groups (est. 2010 [1] ) 92.4% black 3.1% multiracial 2.7% white 1.3% East Indian 0.5% other/unspecified Religion 75.6% Christian 2.5% other 20.6% none 1.2% unspecified [1] Demonym(s) Barbadian Bajan (colloquial) Government Unitary parliamentary constitutional monarchy • Monarch Elizabeth II •  Governor-General Dame Sandra Mason • Prime Minister Mia Mottley Legislature Parliament • Upper house Senate • Lower house House of Assembly Independence • From the United Kingdom 30 November 1966 Area • Total 439 km 2 (169 sq mi) (183rd) • Water (%) Negligib
                    閱讀完整內容

                    How to read a connectionString WITH PROVIDER in .NET Core?

                    圖片
                    6 I added .AddJsonFile("Connections.json", optional: true, reloadOnChange: true) in public Startup(IHostingEnvironment env) Connections.json contains: "ConnectionStrings": "DefaultConnection": "Server=(localdb)\mssqllocaldb;Database=DATABASE;Trusted_Connection=True;MultipleActiveResultSets=true", "COR-W81-101": "Data Source=DATASOURCE;Initial Catalog=P61_CAFM_Basic;User Id=USERID;Password=PASSWORD;Persist Security Info=False;MultipleActiveResultSets=False;Packet Size=4096;", "COR-W81-100": "Data Source=DATASOURCE;Initial Catalog=Post_PS;User Id=USERID;Password=PASSWORD;Persist Security Info=False;MultipleActiveResultSets=False;Packet Size=4096;", "MSEDGEWIN10": "Data Source=DATASOURCE; Initial Catalog=COR_Basic; Persist Security Info=False;Integrated Security=true;MultipleActiveResultSets=False;Packet Size=4096;Application Name="COR_Basic"", "server&qu
                    閱讀完整內容

                    Node.js Script on GitHub Pages or Amazon S3

                    圖片
                    0 I'm wondering if I can run a Node.js script in my Jekyll page from GitHub Pages or Amazon S3? I think it can't run on GitHub Pages since it doesn't support server side code. Not too sure. The code is below: var Airtable = require('airtable'); var base = new Airtable(apiKey: 'YOUR_API_KEY').base('appAnZVyYqusNPV5Q'); base('Invitee list').select( // Selecting the first 3 records in Complete list: maxRecords: 3, view: "Complete list" ).eachPage(function page(records, fetchNextPage) // This function (`page`) will get called for each page of records. records.forEach(function(record) console.log('Retrieved', record.get('Email')); ); // To fetch the next page of records, call `fetchNextPage`. // If there are more records, `page` will get called again. // If there are no more records, `done` will get called. fetchNextPage(); , function done(err) if (err) console.error(err); return; ); node.js
                    閱讀完整內容