News

Today, we’ve shipped a new version 0.2.2 of droonga-client-ruby! It provides client library for Ruby and command line tools.

Most important topic of this version is: now the droonga-groonga command supports multiple commands given via the standard input, like Groonga’s groonga command. For example, you can apply your schema definition to a Droonga cluster like as:

$ cat /path/to/schema.grn | droonga-groonga --host node0 --port 10031

You already been able to get similar result by a combination of grn2drn and the droonga-request command, but now you can do it by only one simple command droonga-groonga easily. Let’s try Droonga via the command, if you are familiar to Groonga!

Moreover, droonga-add, droonga-system-status and droonga-groonga now support a new command line option --dry-run. You’ll do it to forecast what’s happen on the cluster by thoese commands.

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

The reference manual of command line tools is now available!

It covers major command line tools provided by Droonga and related projects, so it will help you to read tutorials and try to Droonga.

Very sorry for this delay, now we’ve released Droonga 1.1.0 with a long-cherished feature!

What’s Droonga?

Droonga is a distributed full text search engine, compatible to Groonga. A Droonga cluster works like an HTTP server compatible to Groonga with the replication feature.

Are you interested in Droonga: how it works and how useful? Let’s try the tutorial for an introduction, and see the overview to understand Droonga’s design strategy more and more.

True hot-add of replica nodes is now available!

The most important topic on this release Droonga 1.1.0 is true hot-add of replica nodes. It was one of long-planned features of Droonga, but we’ve got it. Now you can add new replica nodes to your Droonga cluster without any downtime, even if you don’t stop crawling/adding new data!

To do it, you have to have two or more replicas in the cluster, except the new adding replica node. For more information, see tutorial for adding replica nodes.

Detailed list of improvements

• Droonga-engine 1.1.0
  • General:
    • Graceful stopping/restarting works correctly with continual inpouring messages.
    • A new parameter single_operation is introduced to single step definitions. If you set it to true, messages for the handler will be delivered to just single volume (one of replicas and slices). It is useful for commands which has to be executed only once in a cluster, like system.status.
    • A new parameter use_all_replicas is introduced to single step definitions. If you set it to true, messages for the handler will be delivered to all replica volumes always. It is useful for commands which has to be executed all replicas, like system.statistics.object.count.per-volume.
    • add command now accepts requests with automatically-convertible mismatched type keys. For example, a string key "1" is available for a table with the key type UInt32.
    • dump command: the value of a column referring any record of another table are correctly exported as its key string, instead of object value (it’s invalid value for a message of the add command). As the result, now tables with reference columns are correctly copied between multiple clusters.
    • dump command: records in a table which has only one column _key are exported correctly.
    • dump command: forwarded messages are now have their own date field.
    • Collectors::RecursiveSum is introduced.
    • system.status command: now the reporter node and its internal name is reported as a part of response body.
    • system.statistics.object.count command is now available. It is used by command line utilities internally.
    • system.statistics.object.count.per-volume command is now available. It is useful to confirm equivalence of replicas.
    • system.absorb-data command is now available. It is used by command line utilities internally.
  • Message format:
    • targetRole field is introduced to the envelope. Now you can specify the role of the engine node which can process the messsage. If mismatched role node receives the message, it will be bounced to suitable node automatically if possible.
    • timeout field is introduced to the envelope. Now you can specify time to expire the request, in seconds.
  • Command line utilities:
    • droonga-engine-join and droonga-engine-absorb-data commands work more surely.
    • A new option --verbose is introduced to monitor internal Serf operations, for droonga-engine-join and droonga-engine-absorb-data.
    • A new command line utility droonga-engine-set-role is available (mainly for debugging).
  • Compatibility with Groonga:
    • Better compatibility to Groonga’s delete command:
      • Works correctly for tables with integer key types. The report by funa1g inspired this improvement. Thanks!
      • Accepts requests with automatically-convertible mismatched type keys. For example, a numeric key 1 is available for a table with the key type ShortText.
• Droonga-http-server 1.1.2
  • Bump required version of express-droonga.
• Express-droonga 1.0.9
  • Updated for the latest Droonga Engine 1.1.0.
  • Supports uber-cache 2.0.0.
  • New endpoints are introduced to inspect connections to Droonga Engine nodes.
    • /engines returns the list of currently conencted Droonga Engine nodes.
    • /connections returns detailed status of internal connections to Droonga Engine nodes.
  • Always reconnects to Droonga Engine nodes when there is any change around the Droonga cluster.
  • Query parameters for /droonga/* endpoints are used as the message body.
• Drndump 1.0.1
  • Becomes modulable. You can use internal libraries for other products.
• Drntest 1.2.0
  • Improvements
    • Supported new directives to activate/diactivate completion and validation of messages.
      • #@enable_completion and #@disable_completion controls completion of request messages. Required fields of request messages are automatically completed by default.
      • #@enable_validation and #@disable_validation controls validation of request messages. Messages are validated by default.
    • #@subscribe-until directive, for subscription type commands like dump. You can unsubscribe the next request following to the directive automatically with given timeout, like: #@subscribe-until 10s
    • A NO RESPONSE result is now returned immediately, for a stalled engine process.
    • Invalid format responses for Groonga commands are now ignored.
• Droonga-client-ruby 0.2.1
  • Required fields of input messages are automatically completed by default.
  • Input messages are automatically validated by default.
  • Timeout option for subscription is available. You can unsubscribe subscription automatically with specified timeout, like: client.subscribe(request, :subscription_timeout => 10)
  • New utility commands are available as shorthand.
    • droonga-system-status: allows you to send a system.status request to the cluster easily.
    • droonga-add: allows you to send an add request to the cluster easily.
    • droonga-groonga: works like the groonga command.
  • droonga-send, droonga-request:
    • A new option --[no-]completion is introduced. You should specify --no-completion to send incomplete messages intentionally.
    • A new option --[no-]validation is introduced. You should specify --no-validation to send invalid messages intentionally.
    • A new option --default-dataset is introduced. It is used for sending messages if they have no dataset field.
    • A new option --default-target-role is introduced. It is used for sending messages if they have no targetRole field.
    • The “date” field is filled with the format same to droonga-engine’s internal one like “2015-04-08T06:16:20.571303Z”.

Conclusion

• Droonga 1.1.0 has been released!
• Now you can add new replica to the cluster without any downtime.
• Droonga project will release a new version continuously!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

Thank you for your entry to our conference “Groonga Night 5th” at 2014-11-29!

Droonga 1.0.9 is an extra release. On the session at the conference, we have some troubles. Those issues have been fixed at this release!

Improvements of Droonga 1.0.9

First, now droonga-engine-join, droonga-engine-unjoin, and droonga-engine-absorb-data are runnable on any node. In old versions, those commands had some elusive preconditions, so they were inconvenience a little.

Second, unjoined nodes are correctly invisible from droonga-http-servers. At the previous version, droonga-http-server unexpectedly kept connections to droonga-engine nodes even if they were already unjoined from the cluster. Now droonga-engine nodes manage their relations by unique id for each Droonga cluster, and droonga-http-server can detect which engine node was unjoined by the cluster’s id.

These fixes will let you more easy to try Droonga!

Detailed list of improvements

• Droonga-engine 1.0.9
  • droonga-engine-join, droonga-engine-unjoin, and droonga-engine-absorb-data work on any host. Instead, you have to specify the host name or the IP address of the working host via the --receiver-host option.
  • Clusters are managed with their own unique id. In previous version, a node unjoined from the Droonga cluster is still a member of the Serf cluster, and there is no information that protocol adapter nodes detect which is actual member or not.
• Droonga-http-server 1.1.0 and Express-droonga 1.0.8
  • Connects to active engine nodes of the cluster of the associated engine node correctly. Previous version unexpectedly kept connections to already unjoined engine nodes.

Conclusion

• Droonga 1.0.9 has been released!
• Some inconvenience limitations of cluster management commands are solved.
• droonga-http-server now works as you expected after cluster management operations.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

What’s Droonga?

Droonga is a distributed full text search engine, compatible to Groonga. A Droonga cluster works like an HTTP server compatible to Groonga with the replication feature.

Are you interested in Droonga: how it works and how useful? Let’s try the tutorial for an introduction, and see the overview to understand Droonga’s design strategy more and more.

Improvements in Droonga 1.0.8

This release Droonga 1.0.8 contains many improvements. Major topics are here:

Orchestration of front-end HTTP server nodes

A front-end HTTP server node based on droonga-http-server has become to a member of the Droonga cluster truly.

In old versions, a droonga-http-server node was strongly bound to a droonga-engine node. So, if a droonga-engine node died, droonga-http-server nodes associated to the droonga-engine node also died.

But now, the limitation has been gone. Even if one of droonga-engine node become unavailable, droonga-http-server nodes keep them working with active droonga-engine nodes automatically. Moreover, a droonga-http-server node distributes requests from clients to multiple droonga-engine nodes. In other words, now it works like a simple load balancer.

Better performance for search operations, especially with groupBy (Groonga’s select with drilldown options)

The performance issue around groupBy is solved for replicas with single slices. Moreover, search operations with offset option also optimized.

Benchmark results (comparisons with Groonga) are here:

Conditions:

• There are 1500000 records from Wikipedia (Japanese)
• All requests are full-text search queries for actual page titles, with drilldown, like: /d/select?command_version=2&table=Pages&limit=10&match_columns=title,text&output_columns=snippet_html(title),snippet_html(text),categories,_key&query_flags=NONE&sortby=title&drilldown=categories&drilldown_limit=10&drilldown_output_columns=_id,_key,_nsubrecs&drilldown_sortby=_nsubrecs&query=Wikipedia%3AText+of+GNU+Free+Documentation+License
• Supposed cache hit rate is 50%.
• Measured on physical PCs for development.
  • node0: Ubuntu 14.04LTS, Intel Core i5 M460 2.53GHz, 8GB RAM
  • node1: Ubuntu 14.04LTS, Intel Core i5 650 3.20GHz, 6GB RAM
  • node2: Ubuntu 14.04LTS, Intel Core i5 650 3.20GHz, 8GB RAM
  • client: Ubuntu 14.04LTS, Intel Core i5-4300U vPro 1.90GHz, 4GB RAM

Detailed results are downloadable.

As above, currently throughput performance of single Droonga node is comparable to Groonga. Moreover, throughput limit can be extended by adding more Droonga nodes to the cluster.

On the other hand, there is a little latency basically on Droonga. However, large number accesses turn the advantage around the upper limit of Groonga. If there are too many requests, Droonga can respond quickly than Groonga.

Better compatibility to Groonga

Some minor incompatibilities have been corrected.

• select command:
  • The output_columns option accepts whitespace-separated list of column names (compatible to Groonga’s command_version=1 form.)
  • output_columns=* works correctly for TABLE_NO_KEY tables.
• column_list command:
  • A _key virtual column is correctly appear in the result, for tables with one of flags: TABLE_HASH_KEY, TABLE_PAT_KEY, and TABLE_DAT_KEY.
  • At an index column, its source is now compatible to Groonga’s one.
• table_create command:
  • The key_type parameter becomes required for TABLE_HASH_KEY, TABLE_PAT_KEY, and TABLE_DAT_KEY tables. You’ll get an error response, if the parameter is not given. (Groonga unexpectedly accepts table_create requests without key_type parameter, but it is a known issue.)

Detailed list of all improvements

• Droonga-engine 1.0.8
  • Better compatibility to Groonga’s select, column_list, and table_create commands. (See above)
  • The daemon option is now ignored in the static configuration file. Now, you always have to specify --daemon option for the droonga-engine command to start it as a daemon.
  • The droonga-engine-configure command now shows prompts for all options always.
  • The rate of absorbed records are limited to 100 records per second by default, for droonga-engine-absorb-data and droonga-engine-join commands.
  • droonga-engine-absorb-data and droonga-engine-join commands now report their progress, if possible.
• Droonga-http-server 1.0.9
  • A new --host option is available to restrict the listening IP address. The default value is 0.0.0.0 (meaning listen all IP addresses).
  • A new --cache-ttl-in-seconds option is available to set the time to live of cached responses, in seconds. The default value is 60 (meaning 1 minute).
  • A new --disable-trust-proxy option is available to disable the feature even if it is activated by the static configuration file.
  • A new --document-root option is available to specify the document root. It is the path to the built-in Groonga admin page by default.
  • The daemon option is now ignored in the static configuration file. Now, you always have to specify --daemon option for the droonga-http-server command to start it as a daemon.
  • The droonga-http-server-configure command now shows prompts for all options always.
  • Responses for most commands are never cached. Now, only responses based on search or Groonga’s select commands and the administration page are cached.
  • A new endopoint /cache is introduced to clear all response caches. To clear cached contents, send an HTTP DELETE request to the path.
  • Improvements from express-droonga itself. (See below)
• Express-droonga 1.0.7
  • Supports multiple Droonga Engine nodes as its backends. Now express-droonga can work like a load balancer.
  • The list of connecting Droonga Engine nodes can be automatically updated based on the actual list of active members in the cluster. This feature is activated by the syncHostNames option for the application.droonga() method.
• Drnbench 1.0.4
  • drnbench-request-response
    • Not only top slow requests, but top fast requests are also reported. It will help you to detect “strange good” results from invalid queries or something. The number of reported fast requests can be customized via the new --n-fast-requests option.
    • Virtual clients are working with multiple processes. If there are multiple processors in your computer, drnbench uses them more effectively.
• Drntest 1.1.7 (released at 2014-11-18)
  • Accept virtual column like _key as a part of Groonga’s column_list command response.

Conclusion

• Droonga 1.0.8 has been released!
• Front-end HTTP server nodes are now orchestrated as nodes in the cluster. Droonga clusters now work more robustly.
• The throughput performance is comparable to / better than Groonga now.
• It becomes more compatible to Groonga.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

What’s Droonga?

Droonga is a distributed full text search engine, compatible to Groonga. A Droonga cluster works like an HTTP server compatible to Groonga with the replication feature.

Are you interested in Droonga: how it works and how useful? Let’s try the tutorial for an introduction, and see the overview to understand Droonga’s design strategy more and more.

A fatal issue of service scripts is fixed!

Droonga 1.0.7 is an urgent release. Service scripts introduced at Droonga 1.0.6 had a fatal issue - installed droonga-engine and droonga-http-server services cannot be started after the computer is rebooted. This problem has been fixed on 1.0.7.

Because service scripts are updated, you have to re-install droonga-engine and droonga-http-server by installation scripts like:

# curl https://raw.githubusercontent.com/droonga/droonga-engine/master/install.sh | \
    bash
# curl https://raw.githubusercontent.com/droonga/droonga-http-server/master/install.sh | \
    bash

Other major topics are:

• There is an improvement about compatibility to Groonga: droonga-engine 1.0.7 now supports query_flags of Groonga’s select command, except ALLOW_UPDATE. Droonga can process queries from your application, even if your application uses non-default query_flags.
• Now you can specify the log level of droonga-http-server’s system log. To set log leve, run the droonga-http-server-configure command as root and reconfigure your service.
• A new tutorial for benchmarking of Droonga and Groonga is now available. It describes how to measure (and compare) performance of Groonga and Droonga.

Detailed list of improvements

• Droonga-engine 1.0.7
  • Better compatibility to Groonga: select command now supports query_flags option. Note: ALLOW_UPDATE is ignored even if you specify, because it is not implemented in Droonga yet.
  • saerch command has some improvements.
    • The value false for allowPragma and allowColumn options in query syntax search conditions is correctly applied. In old versions, they options are always true even if you intentionally specified false for them.
    • allowLeadingNot option is available in query syntax search conditions. It is false by default.
  • Works correctly as a service even if you restarted the computer itself.
  • droonga-engine-configure now asks the log level.
• Droonga-http-server 1.0.8
  • Works correctly as a service even if you restarted the computer itself.
  • Log level of the system log is now customizable. You just have to put a line like system_log_level: debug into the droonga-http-server.yaml. And, droonga-http-server-configure also asks the log level.
• Express-droonga 1.0.6
  • Debug logs are now reported via given logger.
• Drnbench 1.0.3
  • drnbench-request-response
    • Accept multiple hosts as a comma separated list via the --default-hosts option. You can simulate load balancing for multiple endpoints easily.
    • Accept plain text file of a list of paths as the patterns file.
  • drnbench-extract-searchterms
    • New --escape option is introduced to escape URI-incompatible characters.

Conclusion

• Droonga 1.0.7 has been released!
• A fatal issue in service scripts is fixed.
• A new tutorial for benchmarking is available.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

Sorry, long time no see. We sadly skipped a release on the last month, however, we’ve been back with Droonga 1.0.6 and a great improvement!

What’s Droonga?

Droonga is a distributed full text search engine, compatible to Groonga. A Droonga cluster works like an HTTP server compatible to Groonga with the replication feature.

Droonga became more easy-to-install!

The most important topic on this release Droonga 1.0.6 is the very easy installation. Droonga’s main components, droonga-engine and droonga-http-server are now installable via script for each, so you just run scripts as the root, just like:

# curl https://raw.githubusercontent.com/droonga/droonga-engine/master/install.sh | \
    bash
# curl https://raw.githubusercontent.com/droonga/droonga-http-server/master/install.sh | \
    bash

Then droonga-engine and droonga-http-server are automatically registered as system services managed via the service command. You just have to run simple commands to start/stop services, like:

# service droonga-engine start
# service droonga-http-server start

Moreover, utility commands to manage nodes in the cluster (like droonga-engine-join and droonga-engine-unjoin) also became more easy-to-use. For more details, see the updated “getting started” tutorial and others.

On the other hand, there is a sad news: Droonga’s installation scripts now work only for Debian, Ubuntu, and CentOS 7. Other environments (for example CentOS 6.5, OS X, BSD… and more) are not supported yet. So, if you have knowledges how to install required commands gem, npm (and git) and how to register custom services, please improve the installation script and send pull requests us!

For developers

The installation script works for not only the latest release, but the latest revision of the master branch also. You just have to give an extra environment variable VERSION, like:

# curl https://raw.githubusercontent.com/droonga/droonga-engine/master/install.sh | \
    VERSION=master bash
# curl https://raw.githubusercontent.com/droonga/droonga-http-server/master/install.sh | \
    VERSION=master bash

Then the script tries to install services based on the master. It will help you to try new fixes before the next release.

And, there is a new tutorial to prepare multiple virtual machines on your own PC. It will help you to try Droonga casually.

Detailed list of improvements

• Droonga-engine 1.0.6
  • The installation script is now available. It automatically installs required softwares and configure the droonga-engine as a system service. Currently it works only for Debian, Ubuntu, and CentOS 7.
  • The service works as a process belonging to a user droonga-engine who is specific for the service. The configuration directory for the service is placed under the home directory of the user.
  • A static configuration file to define default parameters (host and so on) is now available. It must be placed into the directory same to catalog.json. You don’t have to run droonga-engine command with many options, anymore.
  • droonga-engine-join now automatically fetches catalog.json from the specified source replica node. Now you don’t have to copy catalog.json from another node before you run droonga-engine-join anymore.
  • A new catalog plugin is introduced as one of default plugins, to fetch catalog.json from existing cluster. The list of plugins in your catalog.json must include it.
  • A new command line utility droonga-engine-configure is available. It generates the static configuration file, the catalog.json for the service. Moreover, it clears old stored data to make the node empty.
  • Some options for utility commands become optional. Important parameters are automatically detected.
  • Restarts server processes more gracefully.
• Droonga-http-server 1.0.7
  • The installation script is now available. It automatically installs required softwares and configure the droonga-http-server as a system service. Currently it works only for Debian, Ubuntu, and CentOS 7.
  • The service works as a process belonging to a user droonga-http-server who is specific for the service. The configuration directory for the service is placed under the home directory of the user.
  • A static configuration file to define default parameters (port and so on) is now available. It must be placed into the configuration directory specified by the environment variable DROONGA_BASE_DIR. You don’t have to run droonga-http-server command with many options, anymore.
  • A new command line utility droonga-http-server-configure is available. It generates the static configuration file for the service.
  • Cached responses are now returned correctly.
• Express-droonga 1.0.5
  • Return cached responses correctly.
• Drntest 1.1.6
  • Sort dump responses for stable test result.
  • Use --ready-notify-fd option to wait droonga-engine is ready.
• Grn2drn 1.0.4
  • Create output directory automatically.

Conclusion

• Droonga 1.0.6 has been released!
• Droonga became more easy-to-use with simple installation method.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

About Droonga

Droonga is a distributed full text search engine, compatible to Groonga.

It is designed as a scalable data processing engine, based on a stream oriented processing model. In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

Are you interested in Droonga? See overview for more details, and try the tutorial. You’ll understand how Droonga works with these documentations.

About this release

The current Droonga version is 1.0.5.

On this release, some stability issues are fixed. Droonga becomes more safe, more usable. Detailed list of improvements are:

• Droonga-engine 1.0.5
  • Restarts server processes more gracefully.
  • Works with search results with vector reference column values correctly.
  • Messages forwarded to other nodes are always buffered for now.
  • Works again for the case: nWorkers == 0
  • droonga-engine-join: Works correctly and safely for databases with much records.
• Droonga-http-server 1.0.6
  • Provides Groonga’s administration page as the document root (/) experimentally.
  • Supports a new --environment option to override the NODE_ENV environment variable.

And, related projects are also released:

• Droonga-client-ruby 0.1.9
  • droonga-send: Accept default protocol, port, and tag. Now you can omit them for the --server option.
• Drnbench 1.0.1
  • Report throughput more correctly.
  • Report percentages of returned HTTP statuses correctly.
  • Report max elapsed time correctly.
  • Add ability to report slow requests.
  • Accept pattern group specific parameters.

Conclusion

• Droonga 1.0.5 has been released!
• On this version, some stability issues are fixed.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

Today, we’ve released grn2drn 1.0.3.

On this version, grn2drn’s --dataset parameter became optional. If you omit the option, Default is used as the dataset.

And, now a new --encoding option is available for grn2drn. It will help you if you created Groonga’s database with non-UTF-8 encoding (Shift_JIS, EUC-JP, etc.)

About Droonga

Droonga is a distributed full text search engine, compatible to Groonga.

It is designed as a scalable data processing engine, based on a stream oriented processing model. In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

Are you interested in Droonga? See overview for more details, and try the tutorial. You’ll understand how Droonga works with these documentations.

About this release

The current Droonga version is 1.0.4.

The most important topic on this release is that new command line tools to modify cluster composition easily. You don’t need to run various commands again and again anymore. See dump-restore tutorial and another tutorial to add/remove replica.

The release has the following improvements:

• Droonga-engine 1.0.4
  • New command (and plugin) status is now available.
  • New command line tools are available.
    • droonga-engine-join and droonga-engine-unjoin help you to modify cluster composition. See the tutorial to add/remove replica.
    • droonga-engine-absorb-data helps you to duplicate clusters. See the tutorial for dump/restore.
    • droonga-engine-catalog-modify helps you to modify existing catalog.json.
• Express-droonga 1.0.4
  • Supports paths like /droonga/(command name) with GET method.
  • Supports paths like /droonga/(command namespace)/(command name) with GET and POST method.
  • Works with the Express 4.4.4.

Conclusion

• Droonga 1.0.4 has been released!
• With new command line tools, now you can join/unjoin nodes for existing clusters easily.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

About Droonga

Droonga is a distributed full text search engine, compatible to Groonga.

It is designed as a scalable data processing engine, based on a stream oriented processing model. In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

Are you interested in Droonga? See overview for more details, and try the tutorial. You’ll understand how Droonga works with these documentations.

About this release

The current Droonga version is 1.0.3.

The most important topic on this release is that there are many improvements to use a Droonga cluster as a stable service. Now Droonga nodes observe each other and keep the cluster working even if one of replicas is dead. And you can add new replicas, remove existing replicas, and replace replicas with new one, without stopping of the cluster (only for “read-only” requests). For more details, see the dump-restore tutorial and another tutorial to add replica.

The release has the following improvements:

• Droonga-engine 1.0.3
  • Alive monitoring (based on Serf) lands. Now, nodes in a cluster observe each other, and the cluster keeps working, even if one of replicas is dead.
  • New commands to dump whole contents of an existing cluster are available. They are used by [drndump][] internally.
  • The command line tool droonga-catalog-generate is renamed to droonga-engine-catalog-generate.
  • Use Default as the name of the default dataset for a catalog.json, generated by droonga-catalog-generate .
  • The path of the configuration directory is now specified via an environment variable DROONGA_BASE_DIR.
  • Fix incompatibilities of the select Groonga command.
    • The default value of the drilldown_output_columns option becomes same to Groonga’s one.
    • Column values of Time type clumns are returned as float numbers correctly.
    • The message structure of results becomes same to Groonga. In previous version, records are wrongly wrapped in an array.
  • Improve features of the select command..
    • The request parameter "attributes" for "elements" in "output" is now available.
    • The special value "*" for "attributes" in "output" is now available, to export all columns.
  • Server process does shutdown/restart gracefully.
  • Restart itself automatically when the catalog.json is updated.
• Express-droonga 1.0.3
  • Groonga compatible load command with GET method is available.
  • Query parameter group_by[(column name)][attributes] for the REST search command now accepts simple comma-separated string value.
• Droonga-http-server 1.0.5
  • Use Default as the name of the default dataset. It is same to Droonga Engine’s one.
  • Use 10041 as the default port number. It is same to Groonga HTTP server’s one.

And, related projects are also released:

• [Drndump 1.0.0][drndump]: It’s a new face!
  • Initial release.
• Droonga-client-ruby 0.1.7
  • Updated the default port number, same to droonga-engine’s one.
• Drntest 1.1.5
  • Add --timeout option to work on slow environment such as Travis CI.
  • Support tmpfs at /dev/shm.
  • Use the environment variable DROONGA_BASE_DIR.
• Droonga-message-pack-packer-ruby 1.0.1
  • Support packing of Groonga::WGS84GeoPoint and Groonga::TokyoGeoPoint values.

Conclusion

• Droonga 1.0.3 has been released!
• You can add new replicas, remove existing replicas, and replace replicas with new one, without stopping of the Droonga cluster, for read-only requests.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

About Droonga

Droonga is a scalable data processing engine, based on a stream oriented processing model. In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

Are you interested in Droonga? See overview for more details, and try the tutorial. You’ll understand how Droonga works with these documentations.

About this release

The current Droonga version is 1.0.2.

The most important topic on this release is that the core component aka Droonga Engine becomes Fluentd-free. As the result, the project (and the Gem package) fluent-plugin-droonga is renamed to droonga-engine. Of course the compatibility of the Fluentd-protocol is still there.

And, now Droonga has better compatibility to Groonga. Moreover there is a new tutorial for Groonga users aiming to migrate from Groonga to Droonga. Droonga is still under development, but it already can work as a Groonga with the replication feature. Feel free to try it on!

Details of improvements on this release are:

• Droonga-engine 1.0.2 (renamed from “fluent-plugin-droonga”)
  • Becomes Fluentd-free.
  • Supports new --daemon and --pid-file options for the daemon mode.
  • More Groonga-compatible features are available:
    • table_list command
    • column_list command
    • column_remove command
    • column_rename command
    • delete command
    • options for select command
      • filter
      • sortby
      • drilldown
      • drilldown_output_columns
      • drilldown_sortby
      • drilldown_offset
      • drilldown_limit
  • A useful command line tool droonga-catalog-generate is included. It helps you to write your custom catalog.json.
  • Parameters for the search command is validated more strictly.
  • The default port number is changed from 24224 (fluentd’s one) to 10031.
• Express-droonga 1.0.2
  • Works with the Express 4.0.
  • Keeps the process alive even if it is disconnected from the Droonga Engine.
  • Groonga compatible load command with POST method is available. (Note: load command with GET method is not supported yet.)
• Droonga-http-server 1.0.4
  • Works with the Express 4.0.
  • Supports a new --enable-trust-proxy option to run the server behind a reverse proxy.
  • Supports a new --plugins option to choose plugins to be activated.
  • Supports new --daemon and --pid-file options for the daemon mode.

And, related projects are also released:

• Droonga-client-ruby 0.1.6
  • Supports droonga’s protocol.
• Drntest 1.1.4
  • Supports more Groonga-compatible commands available at droonga-engine 1.0.2.

Conclusion

• Droonga 1.0.2 has been released!
• There is better compatibility with Groonga - it’s the time to try Droonga, migrating from Groonga!
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

At the day 2014-03-29, we released grn2drn 1.0.1 also. We forgot to write this topic in the list of updated projects…

The project grn2drn provides some command line tools to use Groonga’s command lists for Droonga. For example, you can generate a JSONs file from Groonga’s load command line.

In this release, we added a new command grn2drn-schema which converts any Groonga’s command list to the “schema” part of the catalog.json. It will help you to write your custom catalog.json by hand, if you know how to define Groonga’s tables.

About Droonga

Droonga is a scalable data processing engine, based on a stream oriented processing model. In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

Are you interested in Droonga? See overview for more details, and try the tutorial. You’ll understand how Droonga works with these documentations.

About this release

The current Droonga version is 1.0.1.

The release has the following improvements:

• Fluent-plugin-droonga 1.0.1
  • More documents around plugin APIs are now available. See the plugin development tutorial and the plugin API reference.
  • Some documented features of the catalog.json are actually implemented. For example:
    • A new vectorOptions option for a schema.
    • New options fact, dimension, nWorkers and so on. For more details, see the reference of the catalog.json.
  • Connections to other Droonga Engine nodes are automatically re-established correctly.
  • Some improvements about the search command
    • The column name _nsubrecs is available as a source with groupBy and sortBy.
    • The element elapsedTime is now available for a value of elements.
    • A new parameter adjusters is introduced. (Not documented yet, so see also Groonga’s document)
    • groupBy becomes faster.
  • And some small bugfixes.
• Express-droonga 1.0.1
  • The bundled HTTP server application is removed. Instead, use another project droonga-http-server.
  • Parallel connections from the Droonga Engine are established correctly.
  • A middleware for response caches is available now. You can see the statistics via /cache/statistics.
  • Query parameters attributes, group_by, and adjusters are available for a REST search command (via /tables/:tableName).
• Droonga-http-server 1.0.3
  • This is an npm package, a frontend for the express-droonga. You don’t have to write your application.js and package.json for general purpose - you just have to install this package and run the droonga-http-server command with options, to start an HTTP protocol adapter on your machine. See also the basic tutorial and the reference manual.

And, related projects are also released:

• Droonga-client-ruby 0.1.5
  • A new command droonga-send is available just for sending requests and ignoring responses.
  • Output results correctly event if there are multiple threads.
• Drntest 1.1.3
  • Supported merging custom catalog.json for each test.
  • Saved actual results on “not checked” cases.
  • Supported a new omit directive.
  • Supported a new require-catalog-version directive.

Conclusion

• Droonga 1.0.1 has been released!
• More documents around plugin APIs are available.
• More features of catalog.json are actually implemented.
• A new npm package droonga-http-server is available as an HTTP protocol adapter frontend. You don’t have to write your application.js and package.json anymore.
• A new droonga-send command is added to droonga-client-ruby.
• Drntest now supports custom catalog.json for each test, and there are more improvements.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

About Droonga

Droonga is a scalable data processing engine, based on a stream oriented processing model.

Droonga is a distributed fulltext search engine

Droonga provides high-power, scalable fulltext search feature including data-storage features.

Droonga can search information from a large database based on multiple partitions and multiple hardwares. You don’t have to think about the maximum limit of the size of storage, CPU resources, and so on. If remaining capacity becomes tight, then you just have to add new nodes to your cluster.

See overview for more details, and try the tutorial.

Droonga is a distributed generic data processing engine

In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

To develop your custom plugins, see the plugin development tutorial.

About this release

The current Droonga version is 1.0.0. This is the first major release!!

The release has the following changes:

• The format of the “catalog.json” is improved and stabilized. More readable, more understandable easily.
• APIs to develop plugins around the handling phase are stabilized. Now you can develop plugins working with the data storage. See the plugin development tutorial for the handling phase for more details.

• Declaration style of plugins is changed a little. Old (Droonga 0.9.9):

  module MyPlugin
    Plugin.registry.register("my-plugin", self)
    ...
  end
  

  New (Droonga 1.0.0):

  module MyPlugin
    extend Plugin
    register("my-plugin")
    ...
  end
  

  The old style still works, but it is deprecated. You should migrate your plugins written for Droonga 0.9.9.

Conclusion

• Droonga 1.0.0 has been released!
• “catalog.json” becomes more readable.
• Plugins can work with the data storage.
• Declaration style of plugins is changed. You should migrate your plugin written for Droonga 0.9.9, as soon as possible.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

About Droonga

Droonga is a scalable data processing engine, based on a stream oriented processing model. In many operations (searching, updating, grouping, and so on), Droonga processes various data by pipeline. As the result, Droonga has large potential around its flexibility and extensibility. Moreover, those features provide high availability for people who develop any data processing engine based on Droonga. You can process complex operations by mixing operations, and you can add custom operations to Droonga via plugins written as Ruby-scripts.

Are you interested in Droonga? See overview for more details, and try the tutorial. You’ll understand how Droonga works with these documentations.

About this release

The current Droonga version is 0.9.9.

The release has the following improvements:

• Plugin APIs for adaptation phase are stabilized.
  • Now you can develop any Droonga plugin which translates request and response messages, between the Droonga Engine and the client including the Protocol Adapter. See the plugin development tutorial for more details.
  • Plugins for other phases will become developable in near future, so please wait for the next release…
• The built-in REST interface of the express-droonga generates queries more intelligently. For example, the search query for the URL “http://localhost:3000/droonga/tables/Store” will be named as “stores” from the table name automatically.

Why we ship Droonga 0.9.9 before 1.0.0?

On the previous release, we announced that we’ll release the version 1.0.0 today. However, now the version 0.9.9 is released. Why not 1.0.0? There are two reasons mainly.

First, plugin APIs were not matured yet. While we were brushing up internal operations of Droonga, we realized that it is hard to develop plugins, because current implementations were built on disorganized codes. Moreover, if people develop any plugin based on the current bad APIs and publish them, then they will become obsolete in very near future, because we define new plugin APIs which completely incompatible with the old one, on the version 1.1, 1.2, or 1.3. We thought that we should stabilize plugin APIs before the version 1.0.0 become public.

Second, we had to rethink the message flow around distributed operations. Droonga should support recursive distributions of messages, but the current message flow couldn’t describe such complex combinations of messages. To introduce such flexibility to Droonga, we had to reconstruct plugin APIs around distributing and collection phases. But rewriting of those mechanism takes much more time and we couldn’t complete it before today.

Then, we decided to postpone shipping of the version 1.0.0. 1.0.0 will be released at 2014-02-28.

Instead, we’ve stabilized plugin APIs for adaptation phase ahead of other phases. Now you can study how to develop Droonga plugins, via steps to develop plugins for adaptation phase.

Conclusion

• Droonga 0.9.9 has been released!
• Now you can develop plugins which translate request and response messages between the Droonga Engine and the client (including the Protocol Adapter). See the plugin development tutorial.
• Plugins for other phases will become developable in the next release, 1.0.0.
• 1.0.0 will be released at 2014-02-28.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

About Droonga

Droonga is a scalable data processing engine. Droonga uses stream oriented processing model. Droonga processes data by pipeline. Many operations such as search, update, group are done in pipeline. The processing model provides flexibility and extensibility. Droonga can also process complex operations by mixing operations. Users can add custom operations to Droonga as Ruby plugins.

Are you interested in Droonga? See overview. It describes more details. Then try tutorial. You will understand how Droonga works with these documentations.

About this release

The current Droonga version is 0.9.0.

The release has the following improvements:

• search: Supported "attributes" for elements of output.
• table_remove: Implemented Groonga compatible table_remove command.
• column_create: Implemented error handling.
• catalog: Supported auto reloading.
• Supported reducing responses from two or more nodes for Groonga compatible commands.
• Supported three or more partitions.
• Supported HTTP streaming.
• droonga-client gem: droonga-client 0.1.1 has been released!
• drntest: Drntest 1.1.1 has been released! Drntest is a testing framework for Droonga.
• drnbench: Drnbench 1.0.0 has been released! Drnbench is a benchmark tool for Droonga.
• chef: Droonga related chef cookbooks are created. It is not well documented but works.

Here are incompatible changes in experimental features:

• watch: Changed command name to watch.publish from watch.notification.

Conclusion

• Droonga 0.9.0 has been released!
• search command is improved. Some unimplemented features are implemented!
• Groonga compatibility is improved.
• Droonga related tools have also been released.
• Droonga plugin will be writable in the next release. Sorry!
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

This is the first stable release of Droonga!

About Droonga

Droonga is a scalable data processing engine. Droonga uses stream oriented processing model. Droonga processes data by pipeline. Many operations such as search, update, group are done in pipeline. The processing model provides flexibility and extensibility. Droonga can also process complex operations by mixing operations. Users can add custom operations to Droonga as Ruby plugins.

Are you interested in Droonga? See overview. It describes more details. Then try tutorial. You will understand how Droonga works with these documentations.

About this release

The current Droonga version is 0.8.0. This is the first stable release.

The release has the following improvements:

• search: Supported groupBy with multiple partitions.
• Supported some error responses.
• Improved plugin API. Plugin API documentation will be published in the next release.
• Added micro seconds information to time value. It uses W3C-DTF format such as 2013-12-29T00:00:00.000000Z.
• catalog: Added version to improve backward compatibility.
• Changed the name of adapter plugin that provides Groonga’s select compatible API to groonga from select. The groonga adapter will provide more Groonga compatible commands.

• Changed plugin path format:

  Old:

  droonga/plugin/#{PLUGIN_TYPE}_#{PLUGIN_NAME}.rb
  

  New:

  droonga/plugin/#{PLUGIN_TYPE}/#{PLUGIN_NAME}.rb
  

  We use a directory per plugin type instead of putting all plugins to droonga/plugin/ directory. Because it is more maintainable.

• Split adapter plugin into input adapter plugin and output adapter plugin. You can in the next release.
• Added documentations in English.

Conclusion

• Droonga 0.8.0 has been released!
• Droonga 0.8.0 is the first stable release.
• Droonga plugin will be writable in the next release.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer! See community to contact us!

This is the first release of Droonga!

About Droonga

Droonga is a scalable data processing engine. Droonga uses stream oriented processing model. Droonga processes data by pipeline. Many operations such as search, update, group are done in pipeline. The processing model provides flexibility and extensibility. Droonga can also process complex operations by mixing operations. Users can add custom operations to Droonga as Ruby plugins.

Are you interested in Droonga? See overview. It describes more details. Then try tutorial. You will understand how Droonga works with these documentations.

Who does try it?

The current Droonga version is 0.7.0. It means “usable” not “stable”. The current Droonga provides basic core features but many other features aren’t implemented yet. Error handlings are also not perfect yet.

If you are interested in Droonga and you don’t care about try & error, please try the current Droonga! If you have questions, please contact us on GitHub issues. Or use mailing list in English (groonga-talk) or in Japanese (groonga-dev). We will help you.

If you are interested in Droonga but you don’t want to avoid try & error, please try again a few months after. Droonga project continues to develop Droonga actively and release a new version every month. You will get more usable Droonga a few months after.

Conclusion

• Droonga 0.7.0 has been released!
• Droonga 0.7.0 is usable but not stable.
• If you’re brave, try Droonga 0.7.0!
• If you’re not so brave, wait a few months.
• Droonga project will release a new version every month!

Droonga project welcomes you to join us as a user and/or a developer!