Kafka Streams application opening too many files on kafka servers









up vote
0
down vote

favorite












I've been working on an application based on the java kafka-streams API, whose goal is to process a stream of data coming from one kafka topic, and produce it into another topic.



As it seems, whenever I start producing messages using the kafka-streams application, file handles just keep opening on the kafka brokers I'm using, and they are never closed, meaning eventually the kafka server ends up with too many open files, and the kafka and zookeeper daemons crash.



I'm using kafka-streams-1.0.1 API jar for Java, and running on JDK 11. The kafka cluster is of Kafka version 1.0.0.



My application's configuration includes the following kafka producer configs:




  • batch.size: set to 100,000 messages.


  • linger.ms: set to 1,000 milliseconds.


  • buffer.memory: set to the byte equivalent of 5 MegaBytes.

The stream processing itself is very simple, and is composed:



stream.map((k,v) -> handle(k,v)).filter((k,v) -> v != null).to(outgoingTopic);


I would appreciate any suggestions you guys might have.






java apache-kafka apache-kafka-streams







share|improve this question





















  • The recommendation by Confluent is to configure the kernel of the host to allow 100.000+ open file handles. Normally the default is way below that.
    – daniu
    yesterday











  • And it is configured to that level. Still, files keep opening
    – Stav Saad
    yesterday










  • I don't think Java 11 is officially supported by Kafka community, or officially tested. Not sure if that's the problem, though
    – cricket_007
    yesterday










  • Java 11 is used as the runtime of my own application and should not affect stuff happening on the Kafka servers
    – Stav Saad
    18 hours ago














up vote
0
down vote

favorite












I've been working on an application based on the java kafka-streams API, whose goal is to process a stream of data coming from one kafka topic, and produce it into another topic.



As it seems, whenever I start producing messages using the kafka-streams application, file handles just keep opening on the kafka brokers I'm using, and they are never closed, meaning eventually the kafka server ends up with too many open files, and the kafka and zookeeper daemons crash.



I'm using kafka-streams-1.0.1 API jar for Java, and running on JDK 11. The kafka cluster is of Kafka version 1.0.0.



My application's configuration includes the following kafka producer configs:




  • batch.size: set to 100,000 messages.


  • linger.ms: set to 1,000 milliseconds.


  • buffer.memory: set to the byte equivalent of 5 MegaBytes.

The stream processing itself is very simple, and is composed:



stream.map((k,v) -> handle(k,v)).filter((k,v) -> v != null).to(outgoingTopic);


I would appreciate any suggestions you guys might have.






java apache-kafka apache-kafka-streams







share|improve this question





















  • The recommendation by Confluent is to configure the kernel of the host to allow 100.000+ open file handles. Normally the default is way below that.
    – daniu
    yesterday











  • And it is configured to that level. Still, files keep opening
    – Stav Saad
    yesterday










  • I don't think Java 11 is officially supported by Kafka community, or officially tested. Not sure if that's the problem, though
    – cricket_007
    yesterday










  • Java 11 is used as the runtime of my own application and should not affect stuff happening on the Kafka servers
    – Stav Saad
    18 hours ago












up vote
0
down vote

favorite









up vote
0
down vote

favorite











I've been working on an application based on the java kafka-streams API, whose goal is to process a stream of data coming from one kafka topic, and produce it into another topic.



As it seems, whenever I start producing messages using the kafka-streams application, file handles just keep opening on the kafka brokers I'm using, and they are never closed, meaning eventually the kafka server ends up with too many open files, and the kafka and zookeeper daemons crash.



I'm using kafka-streams-1.0.1 API jar for Java, and running on JDK 11. The kafka cluster is of Kafka version 1.0.0.



My application's configuration includes the following kafka producer configs:




  • batch.size: set to 100,000 messages.


  • linger.ms: set to 1,000 milliseconds.


  • buffer.memory: set to the byte equivalent of 5 MegaBytes.

The stream processing itself is very simple, and is composed:



stream.map((k,v) -> handle(k,v)).filter((k,v) -> v != null).to(outgoingTopic);


I would appreciate any suggestions you guys might have.






java apache-kafka apache-kafka-streams







share|improve this question













I've been working on an application based on the java kafka-streams API, whose goal is to process a stream of data coming from one kafka topic, and produce it into another topic.



As it seems, whenever I start producing messages using the kafka-streams application, file handles just keep opening on the kafka brokers I'm using, and they are never closed, meaning eventually the kafka server ends up with too many open files, and the kafka and zookeeper daemons crash.



I'm using kafka-streams-1.0.1 API jar for Java, and running on JDK 11. The kafka cluster is of Kafka version 1.0.0.



My application's configuration includes the following kafka producer configs:




  • batch.size: set to 100,000 messages.


  • linger.ms: set to 1,000 milliseconds.


  • buffer.memory: set to the byte equivalent of 5 MegaBytes.

The stream processing itself is very simple, and is composed:



stream.map((k,v) -> handle(k,v)).filter((k,v) -> v != null).to(outgoingTopic);


I would appreciate any suggestions you guys might have.






java apache-kafka apache-kafka-streams




java apache-kafka apache-kafka-streams






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked yesterday









Stav Saad

541413




541413











  • The recommendation by Confluent is to configure the kernel of the host to allow 100.000+ open file handles. Normally the default is way below that.
    – daniu
    yesterday











  • And it is configured to that level. Still, files keep opening
    – Stav Saad
    yesterday










  • I don't think Java 11 is officially supported by Kafka community, or officially tested. Not sure if that's the problem, though
    – cricket_007
    yesterday










  • Java 11 is used as the runtime of my own application and should not affect stuff happening on the Kafka servers
    – Stav Saad
    18 hours ago
















  • The recommendation by Confluent is to configure the kernel of the host to allow 100.000+ open file handles. Normally the default is way below that.
    – daniu
    yesterday











  • And it is configured to that level. Still, files keep opening
    – Stav Saad
    yesterday










  • I don't think Java 11 is officially supported by Kafka community, or officially tested. Not sure if that's the problem, though
    – cricket_007
    yesterday










  • Java 11 is used as the runtime of my own application and should not affect stuff happening on the Kafka servers
    – Stav Saad
    18 hours ago















The recommendation by Confluent is to configure the kernel of the host to allow 100.000+ open file handles. Normally the default is way below that.
– daniu
yesterday





The recommendation by Confluent is to configure the kernel of the host to allow 100.000+ open file handles. Normally the default is way below that.
– daniu
yesterday













And it is configured to that level. Still, files keep opening
– Stav Saad
yesterday




And it is configured to that level. Still, files keep opening
– Stav Saad
yesterday












I don't think Java 11 is officially supported by Kafka community, or officially tested. Not sure if that's the problem, though
– cricket_007
yesterday




I don't think Java 11 is officially supported by Kafka community, or officially tested. Not sure if that's the problem, though
– cricket_007
yesterday












Java 11 is used as the runtime of my own application and should not affect stuff happening on the Kafka servers
– Stav Saad
18 hours ago




Java 11 is used as the runtime of my own application and should not affect stuff happening on the Kafka servers
– Stav Saad
18 hours ago

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53237372%2fkafka-streams-application-opening-too-many-files-on-kafka-servers%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes















 

draft saved


draft discarded















































 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53237372%2fkafka-streams-application-opening-too-many-files-on-kafka-servers%23new-answer', 'question_page');

);

Post as a guest

































































-

這個網誌中的熱門文章

Barbados

圖片
For other uses, see Barbados (disambiguation). Not to be confused with Barbuda. Coordinates: 13°10′N 59°33′W  /  13.167°N 59.550°W  / 13.167; -59.550 Barbados Flag Coat of arms Motto:  "Pride and Industry" Anthem:  In Plenty and In Time of Need Capital and largest city Bridgetown 13°06′N 59°37′W  /  13.100°N 59.617°W  / 13.100; -59.617 Official languages English Recognised regional languages Bajan Creole Ethnic groups (est. 2010 [1] ) 92.4% black 3.1% multiracial 2.7% white 1.3% East Indian 0.5% other/unspecified Religion 75.6% Christian 2.5% other 20.6% none 1.2% unspecified [1] Demonym(s) Barbadian Bajan (colloquial) Government Unitary parliamentary constitutional monarchy • Monarch Elizabeth II •  Governor-General Dame Sandra Mason • Prime Minister Mia Mottley Legislature Parliament • Upper house Senate • Lower house House of Assembly Independence • From the United Kingdom 30 November 1966 Area • Total 439 km 2 (169 sq mi) (183rd) • Water (%) Negligib
閱讀完整內容

How to read a connectionString WITH PROVIDER in .NET Core?

圖片
6 I added .AddJsonFile("Connections.json", optional: true, reloadOnChange: true) in public Startup(IHostingEnvironment env) Connections.json contains: "ConnectionStrings": "DefaultConnection": "Server=(localdb)\mssqllocaldb;Database=DATABASE;Trusted_Connection=True;MultipleActiveResultSets=true", "COR-W81-101": "Data Source=DATASOURCE;Initial Catalog=P61_CAFM_Basic;User Id=USERID;Password=PASSWORD;Persist Security Info=False;MultipleActiveResultSets=False;Packet Size=4096;", "COR-W81-100": "Data Source=DATASOURCE;Initial Catalog=Post_PS;User Id=USERID;Password=PASSWORD;Persist Security Info=False;MultipleActiveResultSets=False;Packet Size=4096;", "MSEDGEWIN10": "Data Source=DATASOURCE; Initial Catalog=COR_Basic; Persist Security Info=False;Integrated Security=true;MultipleActiveResultSets=False;Packet Size=4096;Application Name="COR_Basic"", "server&qu
閱讀完整內容

Node.js Script on GitHub Pages or Amazon S3

圖片
0 I'm wondering if I can run a Node.js script in my Jekyll page from GitHub Pages or Amazon S3? I think it can't run on GitHub Pages since it doesn't support server side code. Not too sure. The code is below: var Airtable = require('airtable'); var base = new Airtable(apiKey: 'YOUR_API_KEY').base('appAnZVyYqusNPV5Q'); base('Invitee list').select( // Selecting the first 3 records in Complete list: maxRecords: 3, view: "Complete list" ).eachPage(function page(records, fetchNextPage) // This function (`page`) will get called for each page of records. records.forEach(function(record) console.log('Retrieved', record.get('Email')); ); // To fetch the next page of records, call `fetchNextPage`. // If there are more records, `page` will get called again. // If there are no more records, `done` will get called. fetchNextPage(); , function done(err) if (err) console.error(err); return; ); node.js
閱讀完整內容