How to get Kafka timestamp for a record in Flink?










1















I have a Flink application that uses the FlinkKafkaConsumer.



I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



How can the timestamp be obtained?






java scala apache-kafka apache-flink kafka-consumer-api







share|improve this question




























    1















    I have a Flink application that uses the FlinkKafkaConsumer.



    I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



    How can the timestamp be obtained?






    java scala apache-kafka apache-flink kafka-consumer-api







    share|improve this question


























      1












      1








      1








      I have a Flink application that uses the FlinkKafkaConsumer.



      I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



      How can the timestamp be obtained?






      java scala apache-kafka apache-flink kafka-consumer-api







      share|improve this question
















      I have a Flink application that uses the FlinkKafkaConsumer.



      I am interested in retrieving the Kafka timestamp for a given record/offset using the KeyedDeserializationSchema. It seems that the topic, partition, offset, and message are available.



      How can the timestamp be obtained?






      java scala apache-kafka apache-flink kafka-consumer-api




      java scala apache-kafka apache-flink kafka-consumer-api






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 16 '18 at 5:57







      codeshark

















      asked Nov 15 '18 at 8:41









      codesharkcodeshark

      1,93673875




      1,93673875






















          1 Answer
          1






          active

          oldest

          votes


















          1














          You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
          The timestamp type is indicated in timestampType().






          share|improve this answer






















            Your Answer






            StackExchange.ifUsing("editor", function ()
            StackExchange.using("externalEditor", function ()
            StackExchange.using("snippets", function ()
            StackExchange.snippets.init();
            );
            );
            , "code-snippets");

            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "1"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













            draft saved

            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53315359%2fhow-to-get-kafka-timestamp-for-a-record-in-flink%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1














            You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
            The timestamp type is indicated in timestampType().






            share|improve this answer



























              1














              You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
              The timestamp type is indicated in timestampType().






              share|improve this answer

























                1












                1








                1







                You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
                The timestamp type is indicated in timestampType().






                share|improve this answer













                You can simply use ConsumerRecord.timestamp() - the timestamp in milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).
                The timestamp type is indicated in timestampType().







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 15 '18 at 10:04









                Axel PodehlAxel Podehl

                1,6861515




                1,6861515





























                    draft saved

                    draft discarded
















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …


                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.

                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53315359%2fhow-to-get-kafka-timestamp-for-a-record-in-flink%23new-answer', 'question_page');

                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    這個網誌中的熱門文章

                    How to read a connectionString WITH PROVIDER in .NET Core?

                    圖片
                    6 I added .AddJsonFile("Connections.json", optional: true, reloadOnChange: true) in public Startup(IHostingEnvironment env) Connections.json contains: "ConnectionStrings": "DefaultConnection": "Server=(localdb)\mssqllocaldb;Database=DATABASE;Trusted_Connection=True;MultipleActiveResultSets=true", "COR-W81-101": "Data Source=DATASOURCE;Initial Catalog=P61_CAFM_Basic;User Id=USERID;Password=PASSWORD;Persist Security Info=False;MultipleActiveResultSets=False;Packet Size=4096;", "COR-W81-100": "Data Source=DATASOURCE;Initial Catalog=Post_PS;User Id=USERID;Password=PASSWORD;Persist Security Info=False;MultipleActiveResultSets=False;Packet Size=4096;", "MSEDGEWIN10": "Data Source=DATASOURCE; Initial Catalog=COR_Basic; Persist Security Info=False;Integrated Security=true;MultipleActiveResultSets=False;Packet Size=4096;Application Name="COR_Basic"", "server&qu
                    閱讀完整內容

                    In R, how to develop a multiplot heatmap.2 figure showing key labels successfully

                    圖片
                    1 2 I'm trying to develop a multiplot heatmap.2 saved to a pdf. I'm having some success but the axis labels are getting chopped off. Subplot titles are also desirable but again the labels are getting chopped. Here's my reproducible code: library(gridExtra) library(grid) library(gridGraphics) library(gplots) Col = colorRampPalette(c("red","orange","yellow", "white")) grab_grob <- function() grid.echo() grid.grab() par(cex.main=0.1, mar = c(1,1,1,1) ) #data<-read.table("heatmap.input.matrix.data.txt") lmat = rbind(c(2,3),c(4,1),c(4,1)) lwid = c(2.5,4) lhei = c(0.5,4,3) labRowvec <- c(rep(NULL, dim(matrix(runif(1000, 1,10),ncol=50))[1])) labColvec <- c(rep(NULL, dim(matrix(runif(1000, 1,10),ncol=50))[2])) gl <- lapply(1:12, function(i) heatmap.2(matrix(runif(1000, 1,10),ncol=50), dendrogram = "none",offsetRow=-0.5, offsetCol=-1,srtCol=0, density="density", lmat =lmat,lhei = l
                    閱讀完整內容

                    Museum of Modern and Contemporary Art of Trento and Rovereto

                    圖片
                    Art museum in Rovereto TN, Italy Museum of Modern and Contemporary Art of Trento and Rovereto Museo d'arte moderna e contemporanea di Trento e Rovereto MART, Entrance Location Corso Angelo Bettini, 43, 38068 Rovereto TN, Italy Coordinates 45°53′38″N 11°02′42″E  /  45.8940°N 11.0450°E  / 45.8940; 11.0450 Coordinates: 45°53′38″N 11°02′42″E  /  45.8940°N 11.0450°E  / 45.8940; 11.0450 Type Art museum Director Gianfranco Maraniello Public transit access Trento train station. Taxis outside station. Website mart.trento.it The Museum of Modern and Contemporary Art of Trento and Rovereto (MART) ( Museo d'Arte Moderna e Contemporanea di Trento e Rovereto , in Italian) is a museum centre in the Italian province of Trento. The main site is in Rovereto, and contains mostly modern and contemporary artworks, including works from renowned Giorgio Morandi, Giorgio de Chirico, Felice Casorati, Carlo Carrà and Fortunato Depero. Fortunato Depero's house in Rovereto (known as Casa d
                    閱讀完整內容