tkrzw-python-0.1.30/ 0000775 0001750 0001750 00000000000 14402705440 013277 5 ustar mikio mikio tkrzw-python-0.1.30/README 0000664 0001750 0001750 00000001070 14402705440 014155 0 ustar mikio mikio ===============================================================
Python interface of Tkrzw
================================================================
Please read the following documents.
COPYING - license
CONTRIBUTING.md - how to contribute
You should install the latest version of Tkrzw to make sure the
compatibility.
To build the library, Python 3.6 or later version is required.
Then, run these commands.
make
make check
sudo make install
See https://dbmx.net/tkrzw/api-python/ for details of the API.
Thanks.
== END OF FILE ==
tkrzw-python-0.1.30/index.rst 0000664 0001750 0001750 00000023635 14402705440 015151 0 ustar mikio mikio Module and Classes
==================
.. autosummary::
:nosignatures:
tkrzw
tkrzw.Utility
tkrzw.Status
tkrzw.StatusException
tkrzw.Future
tkrzw.DBM
tkrzw.Iterator
tkrzw.AsyncDBM
tkrzw.File
Introduction
============
DBM (Database Manager) is a concept to store an associative array on a permanent storage. In other words, DBM allows an application program to store key-value pairs in a file and reuse them later. Each of keys and values is a string or a sequence of bytes. A key must be unique within the database and a value is associated to it. You can retrieve a stored record with its key very quickly. Thanks to simple structure of DBM, its performance can be extremely high.
Tkrzw is a library implementing DBM with various algorithms. It features high degrees of performance, concurrency, scalability and durability. The following data structures are provided.
- HashDBM : File datatabase manager implementation based on hash table.
- TreeDBM : File datatabase manager implementation based on B+ tree.
- SkipDBM : File datatabase manager implementation based on skip list.
- TinyDBM : On-memory datatabase manager implementation based on hash table.
- BabyDBM : On-memory datatabase manager implementation based on B+ tree.
- CacheDBM : On-memory datatabase manager implementation with LRU deletion.
- StdHashDBM : On-memory DBM implementations using std::unordered_map.
- StdTreeDBM : On-memory DBM implementations using std::map.
Whereas Tkrzw is C++ library, this package provides its Python interface. All above data structures are available via one adapter class ":class:`DBM`". Read the `homepage `_ for details.
DBM stores key-value pairs of strings. Each string is represented as bytes in Python. You can specify any type of objects as keys and values if they can be converted into strings, which are "encoded" into bytes. When you retreive the value of a record, the type is determined according to the method: Get for bytes, GetStr for string, or [] for the same type as the key.
Symbols of the module "tkrzw" should be imported in each source file of application programs.::
import tkrzw
An instance of the class ":class:`DBM`" is used in order to handle a database. You can store, delete, and retrieve records with the instance. The result status of each operation is represented by an object of the class ":class:`Status`". Iterator to access access each record is implemented by the class ":class:`Iterator`".
Installation
============
Install the latest version of Tkrzw beforehand and get the package of the Python binding of Tkrzw. Python 3.6 or later is required to use this package.
Enter the directory of the extracted package then perform installation. If your system uses another command than the "python3" command, edit the Makefile beforehand.::
make
make check
sudo make install
Example
=======
The following code is a typical example to use a database. A DBM object can be used like a dictionary object. As DBM implements the generic iterator protocol, you can access each record with the "for" loop.::
import tkrzw
# Prepares the database.
dbm = tkrzw.DBM()
dbm.Open("casket.tkh", True, truncate=True, num_buckets=100)
# Sets records.
# If the operation fails, a runtime exception is raised.
# Keys and values are implicitly converted into bytes.
dbm["first"] = "hop"
dbm["second"] = "step"
dbm["third"] = "jump"
# Retrieves record values.
# If the operation fails, a runtime exception is raised.
# Retrieved values are strings if keys are strings.
print(dbm["first"])
print(dbm["second"])
print(dbm["third"])
try:
print(dbm["fourth"])
except tkrzw.StatusException as e:
print(repr(e))
# Checks and deletes a record.
if "first" in dbm:
del dbm["first"]
# Traverses records.
# Retrieved keys and values are always bytes so we decode them.
for key, value in dbm:
print(key.decode(), value.decode())
# Closes the database.
dbm.Close()
The following code is a more complex example. Resources of DBM and Iterator are bound to their objects so when the refenrece count becomes zero, resources are released. Even if the database is not closed, the destructor closes it implicitly. The method "OrDie" throws an exception on failure so it is useful for checking errors.::
import tkrzw
# Prepares the database.
# Options are given by dictionary expansion.
# All methods except for [] and []= don't raise exceptions.
dbm = tkrzw.DBM()
open_params = {
"max_page_size": 4080,
"max_branches": 256,
"key_comparator": "decimal",
"concurrent": True,
"truncate": True,
}
status = dbm.Open("casket.tkt", True, **open_params)
if not status.IsOK():
raise tkrzw.StatusException(status)
# Sets records.
# The method OrDie raises a runtime error on failure.
dbm.Set(1, "hop").OrDie()
dbm.Set(2, "step").OrDie()
dbm.Set(3, "jump").OrDie()
# Retrieves records without checking errors.
# On failure, the return value is None.
print(dbm.GetStr(1))
print(dbm.GetStr(2))
print(dbm.GetStr(3))
print(dbm.GetStr(4))
# To know the status of retrieval, give a status object to Get.
# You can compare a status object and a status code directly.
status = tkrzw.Status()
value = dbm.GetStr(1, status)
print("status: " + str(status))
if status == tkrzw.Status.SUCCESS:
print("value: " + value)
# Rebuilds the database.
# Almost the same options as the Open method can be given.
dbm.Rebuild(align_pow=0, max_page_size=1024).OrDie()
# Traverses records with an iterator.
iter = dbm.MakeIterator()
iter.First()
while True:
status = tkrzw.Status()
record = iter.GetStr(status)
if not status.IsOK():
break
print(record[0], record[1])
iter.Next()
# Closes the database.
dbm.Close().OrDie()
The following code is a typical example of coroutine usage. The AsyncDBM class manages a thread pool and handles database operations in the background in parallel. Each Method of AsyncDBM returns a Future object to monitor the result. The Future class implements the awaitable protocol so that the instance is usable with the "await" sentence to await the operation while yielding the execution ownership.::
import asyncio
import tkrzw
async def main():
# Prepares the database.
dbm = tkrzw.DBM()
dbm.Open("casket.tkh", True, truncate=True, num_buckets=100)
# Prepares the asynchronous adapter with 4 worker threads.
adbm = tkrzw.AsyncDBM(dbm, 4)
# Executes the Set method asynchronously.
future = adbm.Set("hello", "world")
# Does something in the foreground.
print("Setting a record")
# Checks the result after awaiting the Set operation.
# Calling Future#Get doesn't yield the coroutine ownership.
status = future.Get()
if status != tkrzw.Status.SUCCESS:
print("ERROR: " + str(status))
# Executes the Get method asynchronously.
future = adbm.GetStr("hello")
# Does something in the foreground.
print("Getting a record")
# Awaits the operation while yielding the execution ownership.
await future
status, value = future.Get()
if status == tkrzw.Status.SUCCESS:
print("VALUE: " + value)
# Releases the asynchronous adapter.
adbm.Destruct()
# Closes the database.
dbm.Close()
asyncio.run(main())
The following code uses Process, ProcessMulti, and ProcessEach functions which take callback functions to process the record efficiently. Process is useful to update a record atomically according to the current value. ProcessEach is useful to access every record in the most efficient way.::
import tkrzw
import re
# Opens the database.
dbm = tkrzw.DBM()
dbm.Open("casket.tkh", True, truncate=True, num_buckets=1000)
# Sets records with lambda functions.
dbm.Process("doc-1", lambda key, value: "Tokyo is the capital city of Japan.", True)
dbm.Process("doc-2", lambda key, value: "Is she living in Tokyo, Japan?", True)
dbm.Process("doc-3", lambda key, value: "She must leave Tokyo!", True)
# Lowers record values.
def Lower(key, value):
# If no matching record, None is given as the value.
if not value: return None
# Sets the new value.
# Note that the key and the value are a "bytes" object.
return value.decode().lower()
dbm.Process("doc-1", Lower, True)
dbm.Process("doc-2", Lower, True)
dbm.Process("non-existent", Lower, True)
# Does the same thing with a lambda function.
dbm.Process("doc-3",
lambda key, value: value.decode().lower() if value else None,
True)
# If you don't update the record, set the third parameter to false.
dbm.Process("doc-3", lambda key, value: print(key, value), False)
# Adds multiple records at once.
dbm.ProcessMulti([
("doc-4", lambda key, value: "Tokyo Go!"),
("doc-5", lambda key, value: "Japan Go!")], True)
# Modifies multiple records at once.
dbm.ProcessMulti([("doc-4", Lower), ("doc-5", Lower)], True)
# Checks the whole content.
# This uses an external iterator and is relavively slow.
for key, value in dbm:
print(key.decode(), value.decode())
# Function for word counting.
word_counts = {}
def WordCounter(key, value):
if not key: return
value = value.decode()
words = [x for x in re.split(r"\W+", value) if x]
for word in words:
word_counts[word] = (word_counts.get(word) or 0) + 1
# The second parameter should be false if the value is not updated.
dbm.ProcessEach(WordCounter, False)
print(word_counts)
# Returning False by the callbacks removes the record.
dbm.Process("doc-1", lambda key, value: False, True)
print(dbm.Count())
dbm.ProcessMulti([("doc-2", lambda key, value: False),
("doc-3", lambda key, value: False)], True)
print(dbm.Count())
dbm.ProcessEach(lambda key, value: False, True)
print(dbm.Count())
# Closes the database.
dbm.Close()
Indices and tables
==================
.. toctree::
:maxdepth: 4
:caption: Contents:
tkrzw
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
tkrzw-python-0.1.30/example4.py 0000775 0001750 0001750 00000006011 14402705440 015371 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Example for process methods
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import tkrzw
import re
# Opens the database.
dbm = tkrzw.DBM()
dbm.Open("casket.tkh", True, truncate=True, num_buckets=1000)
# Sets records with lambda functions.
dbm.Process("doc-1", lambda key, value: "Tokyo is the capital city of Japan.", True)
dbm.Process("doc-2", lambda key, value: "Is she living in Tokyo, Japan?", True)
dbm.Process("doc-3", lambda key, value: "She must leave Tokyo!", True)
# Lowers record values.
def Lower(key, value):
# If no matching record, None is given as the value.
if not value: return None
# Sets the new value.
# Note that the key and the value are a "bytes" object.
return value.decode().lower()
dbm.Process("doc-1", Lower, True)
dbm.Process("doc-2", Lower, True)
dbm.Process("non-existent", Lower, True)
# Does the same thing with a lambda function.
dbm.Process("doc-3",
lambda key, value: value.decode().lower() if value else None,
True)
# If you don't update the record, set the third parameter to false.
dbm.Process("doc-3", lambda key, value: print(key, value), False)
# Adds multiple records at once.
dbm.ProcessMulti([
("doc-4", lambda key, value: "Tokyo Go!"),
("doc-5", lambda key, value: "Japan Go!")], True)
# Modifies multiple records at once.
dbm.ProcessMulti([("doc-4", Lower), ("doc-5", Lower)], True)
# Checks the whole content.
# This uses an external iterator and is relavively slow.
for key, value in dbm:
print(key.decode(), value.decode())
# Function for word counting.
word_counts = {}
def WordCounter(key, value):
if not key: return
value = value.decode()
words = [x for x in re.split(r"\W+", value) if x]
for word in words:
word_counts[word] = (word_counts.get(word) or 0) + 1
# The second parameter should be false if the value is not updated.
dbm.ProcessEach(WordCounter, False)
print(word_counts)
# Returning False by the callbacks removes the record.
dbm.Process("doc-1", lambda key, value: False, True)
print(dbm.Count())
dbm.ProcessMulti([("doc-2", lambda key, value: False),
("doc-3", lambda key, value: False)], True)
print(dbm.Count())
dbm.ProcessEach(lambda key, value: False, True)
print(dbm.Count())
# Closes the database.
dbm.Close()
# END OF FILE
tkrzw-python-0.1.30/example2.py 0000775 0001750 0001750 00000004421 14402705440 015372 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Example for basic usage of the hash database
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import tkrzw
# Prepares the database.
# Options are given by dictionary expansion.
# All methods except for [] and []= don't raise exceptions.
dbm = tkrzw.DBM()
open_params = {
"max_page_size": 4080,
"max_branches": 256,
"key_comparator": "decimal",
"concurrent": True,
"truncate": True,
}
status = dbm.Open("casket.tkt", True, **open_params)
if not status.IsOK():
raise tkrzw.StatusException(status)
# Sets records.
# The method OrDie raises a runtime error on failure.
dbm.Set(1, "hop").OrDie()
dbm.Set(2, "step").OrDie()
dbm.Set(3, "jump").OrDie()
# Retrieves records without checking errors.
# On failure, the return value is None.
print(dbm.GetStr(1))
print(dbm.GetStr(2))
print(dbm.GetStr(3))
print(dbm.GetStr(4))
# To know the status of retrieval, give a status object to Get.
# You can compare a status object and a status code directly.
status = tkrzw.Status()
value = dbm.GetStr(1, status)
print("status: " + str(status))
if status == tkrzw.Status.SUCCESS:
print("value: " + value)
# Rebuilds the database.
# Almost the same options as the Open method can be given.
dbm.Rebuild(align_pow=0, max_page_size=1024).OrDie()
# Traverses records with an iterator.
iter = dbm.MakeIterator()
iter.First()
while True:
status = tkrzw.Status()
record = iter.GetStr(status)
if not status.IsOK():
break
print(record[0], record[1])
iter.Next()
# Closes the database.
dbm.Close().OrDie()
# END OF FILE
tkrzw-python-0.1.30/setup.py 0000664 0001750 0001750 00000007765 14402705440 015030 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
# --------------------------------------------------------------------------------------------------
# Building configurations
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
# --------------------------------------------------------------------------------------------------
import os
import platform
import subprocess
import sys
from setuptools import Extension, setup
package_name = "tkrzw"
package_version = "0.1.29"
package_description = "a set of implementations of DBM"
package_author = "Mikio Hirabayashi"
package_author_email = "hirarin@gmail.com"
package_url = "http://dbmx.net/tkrzw/"
module_name = "tkrzw"
keywords = "tkrzw dbm database performance concurrency scalability durability"
classifiers = [
"Development Status :: 4 - Beta",
"Topic :: Database",
"Topic :: Database :: Database Engines/Servers",
"Topic :: Software Development :: Libraries",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: Implementation :: CPython",
]
long_description = """
Python interface of Tkrzw
=========================
[Tkrzw](https://dbmx.net/tkrzw/) is a C++ library implementing DBM with various
algorithms. It features high degrees of performance, concurrency, scalability and
durability.
We build the following wheels:
- musllinux_1_2_x86_64
- manylinux_2_27_x86_64
- TODO: macos
- TODO: windows
Issues
------
- Please report issues related to the build and the wheels to
[tkrzw-wheels](https://github.com/ganwell/tkrzw-wheels)
- Please report issues related to the binding to
[tkrzw-python](https://github.com/estraier/tkrzw-python)
Please read the following documents.
- [COPYING](https://github.com/estraier/tkrzw-python/blob/master/COPYING)
- [CONTRIBUTING.md](https://github.com/estraier/tkrzw-python/blob/master/CONTRIBUTING.md)
""".strip()
extra_compile_args = ["-std=c++17", "-Wall"]
sources = ["tkrzw.cc"]
ld_paths = ["/usr/local/lib"]
os.environ["LD_LIBRARY_PATH"] = os.environ.get("LD_LIBRARY_PATH", "") + ":".join(
ld_paths
)
def parse_build_flags(cmd_args):
result = {}
try:
output = subprocess.check_output(cmd_args).strip().decode()
for item in output.split():
flag = item[0:2]
result.setdefault(flag, []).append(item[2:])
except FileNotFoundError:
sys.stderr.write(f"Warning: {cmd_args[0]} not found")
return result
# Parse include dirs
flags = parse_build_flags(["tkrzw_build_util", "config", "-i"])
include_dirs = flags.get("-I", []) or ["/usr/local/include"]
# Parse library dirs and libraries
flags = parse_build_flags(["tkrzw_build_util", "config", "-l"])
library_dirs = flags.get("-L", ["/usr/local/lib"])
libraries = flags.get("-l", [])
if not libraries:
if platform.system() == "Darwin":
libraries = ["tkrzw", "stdc++", "pthread", "m", "c"]
else:
libraries = ["tkrzw", "stdc++", "rt", "atomic", "pthread", "m", "c"]
setup(
name=package_name,
version=package_version,
description=package_description,
author=package_author,
author_email=package_author_email,
url=package_url,
ext_modules=[
Extension(
module_name,
include_dirs=include_dirs,
extra_compile_args=extra_compile_args,
sources=sources,
library_dirs=library_dirs,
libraries=libraries,
),
],
keywords=keywords,
classifiers=classifiers,
long_description=long_description,
long_description_content_type="text/markdown",
)
# END OF FILE
tkrzw-python-0.1.30/test.py 0000775 0001750 0001750 00000116440 14402705440 014641 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Test cases
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import asyncio
import math
import os
import random
import re
import shutil
import sys
import tempfile
import threading
import time
import unittest
from tkrzw import *
# Unit testing framework.
class TestTkrzw(unittest.TestCase):
# Prepares resources.
def setUp(self):
tmp_prefix = "tkrzw-python-"
self.test_dir = tempfile.mkdtemp(prefix=tmp_prefix)
# Cleanups resources.
def tearDown(self):
shutil.rmtree(self.test_dir)
# Makes a temporary path.
def _make_tmp_path(self, name):
return os.path.join(self.test_dir, name)
# Utility tests.
def testUtility(self):
self.assertTrue(re.search(r"^\d+.\d+.\d+$", Utility.VERSION))
self.assertTrue(len(Utility.OS_NAME))
self.assertTrue(Utility.PAGE_SIZE > 0)
self.assertEqual(-2 ** 31, Utility.INT32MIN)
self.assertEqual(2 ** 31 - 1, Utility.INT32MAX)
self.assertEqual(2 ** 32 - 1, Utility.UINT32MAX)
self.assertEqual(-2 ** 63, Utility.INT64MIN)
self.assertEqual(2 ** 63 - 1, Utility.INT64MAX)
self.assertEqual(2 ** 64 - 1, Utility.UINT64MAX)
if Utility.OS_NAME == "Linux":
self.assertTrue(Utility.GetMemoryCapacity() > 0)
self.assertTrue(Utility.GetMemoryUsage() > 0)
self.assertTrue(3042090208, Utility.PrimaryHash("abc", (1 << 32) - 1))
self.assertTrue(16973900370012003622, Utility.PrimaryHash("abc"))
self.assertTrue(702176507, Utility.SecondaryHash("abc", (1 << 32) - 1))
self.assertTrue(1765794342254572867, Utility.SecondaryHash("abc"))
self.assertEqual(0, Utility.EditDistanceLev("", ""))
self.assertEqual(1, Utility.EditDistanceLev("ac", "abc"))
self.assertEqual(1, Utility.EditDistanceLev("あいう", "あう"))
# Status tests.
def testStatus(self):
status = Status()
self.assertEqual(Status.SUCCESS, status)
self.assertEqual(Status.SUCCESS, status.GetCode())
self.assertEqual("", status.GetMessage())
self.assertTrue(status.IsOK())
status.Set(Status.NOT_FOUND_ERROR, "foobar")
self.assertEqual(Status.NOT_FOUND_ERROR, status)
self.assertEqual("NOT_FOUND_ERROR: foobar", str(status))
self.assertTrue("foobar" in repr(status))
self.assertFalse(status.IsOK())
s2 = Status(Status.NOT_IMPLEMENTED_ERROR, "void")
status.Join(s2)
self.assertEqual("NOT_FOUND_ERROR: foobar", str(status))
status.Set(Status.SUCCESS, "OK")
status.Join(s2)
self.assertEqual("NOT_IMPLEMENTED_ERROR: void", str(status))
try:
status.OrDie()
except StatusException as e:
self.assertTrue("void" in str(e))
else:
self.fail("no exception")
self.assertEqual("SUCCESS", Status.CodeName(Status.SUCCESS))
self.assertEqual("INFEASIBLE_ERROR", Status.CodeName(Status.INFEASIBLE_ERROR))
# Basic tests.
def testBasic(self):
confs = [
{"path": "casket.tkh",
"open_params":
{"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100,
"fbp_capacity": 64, "concurrent": True},
"rebuild_params":
{"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100,
"fbp_capacity": 64},
"synchronize_params": {},
"expected_class": "HashDBM"},
{"path": "casket.tkt",
"open_params":
{"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100,
"fbp_capacity": 64, "max_page_size": 100, "max_branches": 4, "max_cached_pages": 1,
"key_comparator": "decimal", "concurrent": True},
"rebuild_params":
{"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100,
"fbp_capacity": 64, "max_page_size": 100, "max_branches": 4, "max_cached_pages": 1},
"synchronize_params": {},
"expected_class": "TreeDBM"},
{"path": "casket.tks",
"open_params":
{"offset_width": 3, "step_unit": 2, "max_level": 3, "sort_mem_size": 100,
"insert_in_order": False, "max_cached_records": 8, "concurrent": True},
"rebuild_params":
{"offset_width": 3, "step_unit": 2, "max_level": 3, "sort_mem_size": 100,
"max_cached_records": 8},
"synchronize_params": {"reducer": "last"},
"expected_class": "SkipDBM"},
{"path": "casket.tiny",
"open_params": {"num_buckets": 10},
"rebuild_params": {"num_buckets": 10},
"synchronize_params": {},
"expected_class": "TinyDBM"},
{"path": "casket.baby",
"open_params": {"key_comparator": "decimal"},
"rebuild_params": {},
"synchronize_params": {},
"expected_class": "BabyDBM"},
{"path": "casket.cache",
"open_params": {"cap_rec_num": 10000, "cap_mem_size": 1000000},
"rebuild_params": {"cap_rec_num": 10000},
"synchronize_params": {},
"expected_class": "CacheDBM"},
{"path": "casket.stdhash",
"open_params": {"num_buckets": 10},
"rebuild_params": {},
"synchronize_params": {},
"expected_class": "StdHashDBM"},
{"path": "casket.stdtree",
"open_params": {},
"rebuild_params": {},
"synchronize_params": {},
"expected_class": "StdTreeDBM"},
{"path": "casket",
"open_params":
{"num_shards": 4, "dbm": "hash", "num_buckets": 100},
"rebuild_params": {},
"synchronize_params": {},
"expected_class": "HashDBM"},
]
for conf in confs:
path = conf["path"]
path = self._make_tmp_path(path) if path else ""
dbm = DBM()
self.assertEqual(0, len(dbm))
open_params = conf["open_params"].copy()
open_params["truncate"] = True
self.assertFalse(dbm.IsOpen())
self.assertEqual(Status.SUCCESS, dbm.Open(path, True, **open_params))
self.assertTrue(dbm.IsOpen())
inspect = dbm.Inspect()
class_name = inspect["class"]
self.assertEqual(conf["expected_class"], class_name)
for i in range(0, 20):
key = "{:08d}".format(i)
value = "{:d}".format(i)
self.assertEqual(Status.SUCCESS, dbm.Set(key, value, False))
for i in range(0, 20, 2):
key = "{:08d}".format(i)
self.assertEqual(Status.SUCCESS, dbm.Remove(key))
if class_name in ("HashDBM", "TreeDBM", "TinyDBM", "BabyDBM"):
self.assertEqual(Status.SUCCESS, dbm.Set("日本", "東京"))
self.assertEqual("東京", dbm.GetStr("日本"))
self.assertEqual(Status.SUCCESS, dbm.Remove("日本"))
self.assertEqual(Status.SUCCESS, dbm.Synchronize(False, **conf["synchronize_params"]))
self.assertEqual(10, dbm.Count())
self.assertTrue(dbm.GetFileSize() > 0)
if path:
self.assertTrue(path in dbm.GetFilePath())
self.assertTrue(dbm.GetTimestamp() > 0)
self.assertTrue(dbm.IsWritable())
self.assertTrue(dbm.IsHealthy())
if class_name in ("TreeDBM", "SkipDBM", "BabyDBM", "StdTreeDBM"):
self.assertTrue(dbm.IsOrdered())
else:
self.assertFalse(dbm.IsOrdered())
self.assertEqual(10, len(dbm))
self.assertTrue(conf["expected_class"] in repr(dbm))
self.assertTrue(conf["expected_class"] in str(dbm))
for i in range(0, 20):
key = "{:08d}".format(i)
value = "new-{:d}".format(i)
status = dbm.Set(key, value, False)
if i % 2 == 0:
self.assertTrue(status == Status.SUCCESS)
else:
self.assertTrue(status == Status.DUPLICATION_ERROR)
sv = dbm.SetAndGet("98765", "apple", False)
self.assertEqual(Status.SUCCESS, sv[0])
self.assertEqual(None, sv[1])
if class_name in ("TreeDBM", "TreeDBM", "TinyDBM", "BabyDBM"):
sv = dbm.SetAndGet("98765", "orange", False)
self.assertEqual(Status.DUPLICATION_ERROR, sv[0])
self.assertEqual("apple", sv[1])
sv = dbm.SetAndGet("98765", b"orange", True)
self.assertEqual(Status.SUCCESS, sv[0])
self.assertEqual(b"apple", sv[1])
self.assertEqual("orange", dbm.GetStr("98765"))
sv = dbm.RemoveAndGet("98765")
self.assertEqual(Status.SUCCESS, sv[0])
self.assertEqual("orange", sv[1])
sv = dbm.RemoveAndGet("98765")
self.assertEqual(Status.NOT_FOUND_ERROR, sv[0])
self.assertEqual(None, sv[1])
self.assertEqual(Status.SUCCESS, dbm.Set("98765", "banana"))
self.assertEqual(Status.SUCCESS, dbm.Remove("98765"))
self.assertEqual(Status.SUCCESS, dbm.Synchronize(False, **conf["synchronize_params"]))
records = {}
for i in range(0, 20):
key = "{:08d}".format(i)
value = "new-{:d}".format(i) if i % 2 == 0 else "{:d}".format(i)
self.assertEqual(value.encode(), dbm.Get(key.encode()))
self.assertEqual(value.encode(), dbm.Get(key))
self.assertEqual(value, dbm.GetStr(key.encode()))
self.assertEqual(value, dbm.GetStr(key))
status = Status()
rec_value = dbm.Get(key.encode(), status)
self.assertEqual(value.encode(), rec_value)
self.assertEqual(Status.SUCCESS, status)
records[key] = value
self.assertEqual(Status.SUCCESS, dbm.Rebuild(**conf["rebuild_params"]))
it_records = {}
iter = dbm.MakeIterator()
status = Status()
record = iter.Get(status)
self.assertEqual(Status.NOT_FOUND_ERROR, status)
self.assertEqual(None, record)
self.assertEqual(Status.SUCCESS, iter.First())
self.assertTrue("0000" in repr(iter))
self.assertTrue("0000" in str(iter))
while True:
status = Status()
record = iter.Get(status)
if status != Status.SUCCESS:
self.assertEqual(Status.NOT_FOUND_ERROR, status)
break
self.assertEqual(2, len(record))
it_records[record[0].decode()] = record[1].decode()
record_str = iter.GetStr(status)
self.assertEqual(Status.SUCCESS, status)
self.assertEqual(2, len(record_str))
self.assertEqual(record[0].decode(), record_str[0])
self.assertEqual(record[1].decode(), record_str[1])
self.assertEqual(record_str[0], iter.GetKeyStr())
self.assertEqual(record_str[1], iter.GetValueStr())
self.assertEqual(Status.SUCCESS, iter.Next())
self.assertEqual(records, it_records)
it_records = {}
for key, value in dbm:
it_records[key.decode()] = value.decode()
self.assertEqual(records, it_records)
self.assertEqual(Status.SUCCESS, iter.Jump("00000011"))
self.assertEqual("00000011", iter.GetKey().decode())
self.assertEqual("11", iter.GetValue().decode())
if dbm.IsOrdered():
self.assertEqual(Status.SUCCESS, iter.Last())
it_records = {}
while True:
status = Status()
record = iter.Get(status)
if status != Status.SUCCESS:
self.assertEqual(Status.NOT_FOUND_ERROR, status)
break
self.assertEqual(2, len(record))
it_records[record[0].decode()] = record[1].decode()
self.assertEqual(Status.SUCCESS, iter.Previous())
self.assertEqual(records, it_records)
if path:
base, ext = os.path.splitext(path)
copy_path = base + "-copy" + ext
self.assertEqual(Status.SUCCESS, dbm.CopyFileData(copy_path))
copy_dbm = DBM()
if "." in path:
self.assertEqual(Status.SUCCESS, copy_dbm.Open(copy_path, False))
else:
params = {"dbm": conf["expected_class"]}
if "num_shards" in open_params:
params["num_shards"] = 0
self.assertEqual(Status.SUCCESS, copy_dbm.Open(copy_path, False, **params))
self.assertTrue(copy_dbm.IsHealthy())
it_records = {}
for key, value in copy_dbm:
it_records[key.decode()] = value.decode()
self.assertEqual(records, it_records)
self.assertEqual(Status.SUCCESS, copy_dbm.Close())
if class_name in ("HashDBM", "TreeDBM"):
restored_path = copy_path + "-restored"
self.assertEqual(Status.SUCCESS, DBM.RestoreDatabase(
copy_path, restored_path, class_name, -1))
export_dbm = DBM()
self.assertEqual(Status.SUCCESS, export_dbm.Open("", True, dbm="BabyDBM"))
self.assertEqual(Status.SUCCESS, dbm.Export(export_dbm))
it_records = {}
for key, value in export_dbm:
it_records[key.decode()] = value.decode()
self.assertEqual(records, it_records)
self.assertEqual(Status.SUCCESS, export_dbm.Clear())
self.assertEqual(0, len(export_dbm))
self.assertEqual(Status.SUCCESS, export_dbm.Set("1", "100"))
value = export_dbm.Increment("10000", 2, 10000, status)
self.assertEqual(Status.SUCCESS, status)
self.assertEqual(value, 10002)
value = export_dbm.Increment("10000", Utility.INT64MIN, 0, status)
self.assertEqual(Status.SUCCESS, status)
self.assertEqual(value, 10002)
self.assertEqual(Status.DUPLICATION_ERROR, export_dbm.Set("1", "101", False))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", "100", "101"))
value = export_dbm.Increment("10000", 2)
self.assertEqual(value, 10004)
self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchange("1", "100", "101"))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", "101", None))
value = export_dbm.Get("1", status)
self.assertEqual(Status.NOT_FOUND_ERROR, status)
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", None, "zzz"))
self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchange("1", None, "yyy"))
self.assertEqual("zzz", export_dbm.GetStr("1", status))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", "zzz", None))
self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchange(
"xyz", DBM.ANY_DATA, DBM.ANY_DATA))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("xyz", None, "abc"))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange(
"xyz", DBM.ANY_DATA, DBM.ANY_DATA))
self.assertEqual("abc", export_dbm.GetStr("xyz", status))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange(
"xyz", DBM.ANY_DATA, "def"))
self.assertEqual("def", export_dbm.GetStr("xyz", status))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange(
"xyz", DBM.ANY_DATA, None))
self.assertTrue(export_dbm.GetStr("xyz", status) == None)
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti(
(("hop", None), ("step", None)),
(("hop", "one"), ("step", "two"))))
self.assertEqual("one", export_dbm.GetStr("hop"))
self.assertEqual("two", export_dbm.GetStr("step"))
status, value = export_dbm.CompareExchangeAndGet("xyz", None, "123");
self.assertEqual(Status.SUCCESS, status)
self.assertEqual(None, value)
status, value = export_dbm.CompareExchangeAndGet("xyz", "123", DBM.ANY_DATA);
self.assertEqual(Status.SUCCESS, status)
self.assertEqual("123", value)
status, value = export_dbm.CompareExchangeAndGet("xyz", DBM.ANY_DATA, None);
self.assertEqual(Status.SUCCESS, status)
self.assertEqual(b"123", value)
self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchangeMulti(
(("hop", "one"), ("step", None)),
(("hop", "uno"), ("step", "dos"))))
self.assertEqual("one", export_dbm.GetStr("hop"))
self.assertEqual("two", export_dbm.GetStr("step"))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti(
(("hop", "one"), ("step", "two")),
(("hop", "1"), ("step", "2"))))
self.assertEqual("1", export_dbm.GetStr("hop"))
self.assertEqual("2", export_dbm.GetStr("step"))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti(
(("hop", "1"), ("step", "2")),
(("hop", None), ("step", None))))
self.assertEqual(None, export_dbm.GetStr("hop"))
self.assertEqual(None, export_dbm.GetStr("step"))
self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchangeMulti(
[("xyz", DBM.ANY_DATA)], [("xyz", "abc")]))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti(
[("xyz", None)], [("xyz", "abc")]))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti(
[("xyz", DBM.ANY_DATA)], [("xyz", "def")]))
self.assertEqual("def", export_dbm.GetStr("xyz"))
self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti(
[("xyz", DBM.ANY_DATA)], [("xyz", None)]))
self.assertEqual(None, export_dbm.GetStr("xyz"))
export_iter = export_dbm.MakeIterator()
self.assertEqual(Status.SUCCESS, export_iter.First())
self.assertEqual(Status.SUCCESS, export_iter.Set("foobar"))
self.assertEqual(Status.SUCCESS, export_iter.Remove())
self.assertEqual(0, len(export_dbm))
self.assertEqual(Status.SUCCESS, export_dbm.Append("foo", "bar", ","))
self.assertEqual(Status.SUCCESS, export_dbm.Append("foo", "baz", ","))
self.assertEqual(Status.SUCCESS, export_dbm.Append("foo", "qux", ""))
self.assertEqual("bar,bazqux", export_dbm.GetStr("foo"))
export_dbm["abc"] = "defg"
self.assertEqual("defg", export_dbm["abc"])
self.assertTrue("abc" in export_dbm)
del export_dbm["abc"]
try:
export_dbm["abc"]
except StatusException as e:
self.assertEqual(Status.NOT_FOUND_ERROR, e.GetStatus())
self.assertFalse("abc" in export_dbm)
self.assertEqual(Status.SUCCESS, export_dbm.SetMulti(True, one="first", two="second"))
self.assertEqual(Status.SUCCESS, export_dbm.AppendMulti(":", one="1", two="2"))
ret_records = export_dbm.GetMulti("one", "two", "three")
self.assertEqual("first:1".encode(), ret_records.get("one".encode()))
self.assertEqual("second:2".encode(), ret_records.get("two".encode()))
self.assertEqual(None, ret_records.get("third".encode()))
ret_records = export_dbm.GetMultiStr("one", "two", "three")
self.assertEqual("first:1", ret_records.get("one"))
self.assertEqual("second:2", ret_records.get("two"))
self.assertEqual(None, ret_records.get("third"))
self.assertEqual(Status.SUCCESS, export_dbm.RemoveMulti("one", "two"))
self.assertEqual(Status.NOT_FOUND_ERROR, export_dbm.RemoveMulti("two", "three"))
status = Status()
self.assertEqual(None, export_dbm.Get("one", status))
self.assertEqual(Status.NOT_FOUND_ERROR, status)
status = Status()
self.assertEqual(None, export_dbm.Get("two", status))
self.assertEqual(Status.NOT_FOUND_ERROR, status)
status = Status()
self.assertEqual(None, export_dbm.Get("three", status))
self.assertEqual(Status.NOT_FOUND_ERROR, status)
self.assertEqual(Status.SUCCESS, export_dbm.Set("zero", "foo"))
self.assertEqual(Status.SUCCESS, export_dbm.Rekey("zero", "one", True))
self.assertEqual(None, export_dbm.Get("zero"))
self.assertEqual("foo", export_dbm.GetStr("one"))
step_count = 0
self.assertEqual(Status.SUCCESS, export_iter.First())
while True:
status.Set(Status.UNKNOWN_ERROR)
record = export_iter.Step(status)
if not record:
self.assertEqual(Status.NOT_FOUND_ERROR, status)
break
self.assertEqual(Status.SUCCESS, status)
step_count += 1
self.assertEqual(export_dbm.Count(), step_count)
step_count = 0
self.assertEqual(Status.SUCCESS, export_iter.First())
while True:
status.Set(Status.UNKNOWN_ERROR)
record = export_iter.StepStr(status)
if not record:
self.assertEqual(Status.NOT_FOUND_ERROR, status)
break
self.assertEqual(Status.SUCCESS, status)
step_count += 1
self.assertEqual(export_dbm.Count(), step_count)
pop_count = 0
while True:
status.Set(Status.UNKNOWN_ERROR)
record = export_dbm.PopFirst(status)
if not record:
self.assertEqual(Status.NOT_FOUND_ERROR, status)
break
self.assertEqual(Status.SUCCESS, status)
pop_count += 1
self.assertEqual(step_count, pop_count)
self.assertEqual(0, export_dbm.Count())
self.assertEqual(Status.SUCCESS, export_dbm.SetMulti(
False, japan="tokyo", china="beijing", korea="seoul", france="paris"))
pop_count = 0
while True:
status.Set(Status.UNKNOWN_ERROR)
record = export_dbm.PopFirstStr(status)
if not record:
self.assertEqual(Status.NOT_FOUND_ERROR, status)
break
self.assertEqual(Status.SUCCESS, status)
pop_count += 1
self.assertEqual(4, pop_count)
self.assertEqual(Status.SUCCESS, export_dbm.PushLast("foo", 0))
record = export_dbm.PopFirst()
self.assertEqual(record[0], b"\0\0\0\0\0\0\0\0")
self.assertEqual(record[1], b"foo")
self.assertEqual(Status.SUCCESS, export_dbm.Close())
self.assertEqual(Status.SUCCESS, dbm.Close())
if path:
self.assertEqual(Status.SUCCESS, dbm.Open(path, False, **open_params))
self.assertEqual(Status.SUCCESS, dbm.Close())
# Basic process-related functions.
def testProcess(self):
path = self._make_tmp_path("casket.tkh")
dbm = DBM()
self.assertEqual(Status.SUCCESS, dbm.Open(path, True, truncate=True, num_buckets=1000))
self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: None, True))
self.assertEqual(None, dbm.GetStr("abc"))
self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: False, True))
self.assertEqual(None, dbm.GetStr("abc"))
self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: "ABCDE", True))
self.assertEqual("ABCDE", dbm.GetStr("abc"))
def Processor1(k, v):
self.assertEqual(b"abc", k)
self.assertEqual(b"ABCDE", v)
return None
self.assertEqual(Status.SUCCESS, dbm.Process("abc", Processor1, False))
self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: False, True))
self.assertEqual(None, dbm.GetStr("abc"))
def Processor2(k, v):
self.assertEqual(b"abc", k)
self.assertEqual(None, v)
return None
self.assertEqual(Status.SUCCESS, dbm.Process("abc", Processor2, False))
for i in range(10):
self.assertEqual(Status.SUCCESS, dbm.Process(i, lambda k, v: i * i, True))
self.assertEqual(10, dbm.Count())
counters = {"empty": 0, "full": 0}
def Processor3(k, v):
if k:
counters["full"] += 1
self.assertEqual(int(k) ** 2, int(v))
else:
counters["empty"] += 1
return None
self.assertEqual(Status.SUCCESS, dbm.ProcessEach(Processor3, False))
self.assertEqual(2, counters["empty"])
self.assertEqual(10, counters["full"])
def Processor4(k, v):
if not k: return
return int(int(v) ** 0.5)
self.assertEqual(Status.SUCCESS, dbm.ProcessEach(Processor4, True))
def Processor5(k, v):
if not k: return
self.assertEqual(int(k), int(v))
return False
self.assertEqual(Status.SUCCESS, dbm.ProcessEach(Processor5, True))
self.assertEqual(0, dbm.Count())
ops = [
("one", lambda key, value: "hop"),
("two", lambda key, value: "step"),
("three", lambda key, value: "jump"),
]
self.assertEqual(Status.SUCCESS, dbm.ProcessMulti(ops, True))
ops = [
("one", lambda key, value: False),
("two", lambda key, value: False),
("three", lambda key, value: value.decode() * 2 if value else "x"),
("four", lambda key, value: value.decode() * 2 if value else "x"),
("three", lambda key, value: value.decode() * 2 if value else "x"),
("four", lambda key, value: value.decode() * 2 if value else "x"),
]
self.assertEqual(Status.SUCCESS, dbm.ProcessMulti(ops, True))
self.assertEqual(2, dbm.Count())
self.assertEqual(None, dbm.GetStr("one"))
self.assertEqual(None, dbm.GetStr("two"))
self.assertEqual("jumpjumpjumpjump", dbm.GetStr("three"))
self.assertEqual("xx", dbm.GetStr("four"))
self.assertEqual(Status.SUCCESS, dbm.Close())
# Thread tests.
def testThread(self):
dbm = DBM()
self.assertEqual(Status.SUCCESS, dbm.Open(
"casket.tkh", True, truncate=True, num_buckets=1000))
rnd_state = random.Random()
num_records = 5000
num_threads = 5
records = {}
class Task(threading.Thread):
def __init__(self, test, thid):
threading.Thread.__init__(self)
self.thid = thid
self.test = test
def run(self):
for i in range(0, num_records):
key_num = rnd_state.randint(1, num_records)
key_num = key_num - key_num % num_threads + self.thid
key = str(key_num)
value = str(key_num * key_num)
if rnd_state.randint(0, num_records) == 0:
self.test.assertEqual(Status.SUCCESS, dbm.Rebuild())
elif rnd_state.randint(0, 10) == 0:
iter = dbm.MakeIterator()
iter.Jump(key)
status = Status()
record = iter.Get(status)
if status == Status.SUCCESS:
self.test.assertEqual(2, len(record))
self.test.assertEqual(key, record[0].decode())
self.test.assertEqual(value, record[1].decode())
status = iter.Next()
self.test.assertTrue(status == Status.SUCCESS or status == Status.NOT_FOUND_ERROR)
elif rnd_state.randint(0, 4) == 0:
status = Status()
rec_value = dbm.Get(key, status)
if status == Status.SUCCESS:
self.test.assertEqual(value, rec_value.decode())
else:
self.test.assertEqual(Status.NOT_FOUND_ERROR, status)
elif rnd_state.randint(0, 4) == 0:
status = dbm.Remove(key)
if status == Status.SUCCESS:
del records[key]
else:
self.test.assertEqual(Status.NOT_FOUND_ERROR, status)
else:
overwrite = rnd_state.randint(0, 2) == 0
status = dbm.Set(key, value, overwrite)
if status == Status.SUCCESS:
records[key] = value
else:
self.test.assertEqual(Status.DUPLICATION_ERROR, status)
if rnd_state.randint(0, 10) == 0:
time.sleep(0.00001)
threads = []
for thid in range(0, num_threads):
threads.append(Task(self, thid))
for th in threads:
th.start()
for th in threads:
th.join()
it_records = {}
for key, value in dbm:
it_records[key.decode()] = value.decode()
self.assertEqual(records, it_records)
self.assertEqual(Status.SUCCESS, dbm.Close())
# Search tests.
def testSearch(self):
confs = [
{"path": "casket.tkh",
"open_params": {"num_buckets": 100}},
{"path": "casket.tkt",
"open_params": {"num_buckets": 100}},
{"path": "casket.tks",
"open_params": {"max_level": 8}},
{"path": "",
"open_params": {"dbm": "TinyDBM", "num_buckets": 100}},
{"path": "",
"open_params": {"dbm": "BabyDBM"}},
]
for conf in confs:
path = conf["path"]
path = self._make_tmp_path(path) if path else ""
dbm = DBM()
open_params = conf["open_params"].copy()
open_params["truncate"] = True
self.assertEqual(Status.SUCCESS, dbm.Open(path, True, **open_params))
for i in range(1, 101):
key = "{:08d}".format(i)
value = "{:d}".format(i)
self.assertEqual(Status.SUCCESS, dbm.Set(key, value, False))
self.assertEqual(Status.SUCCESS, dbm.Synchronize(False))
self.assertEqual(100, dbm.Count())
self.assertEqual(12, len(dbm.Search("contain", "001")))
self.assertEqual(3, len(dbm.Search("contain", "001", 3)))
self.assertEqual(10, len(dbm.Search("begin", "0000001")))
self.assertEqual(10, len(dbm.Search("end", "1")))
self.assertEqual(10, len(dbm.Search("regex", r"^\d+1$")))
self.assertEqual(10, len(dbm.Search("regex", r"^\d+1$", 0,)))
self.assertEqual(3, len(dbm.Search("edit", "00000100", 3)))
self.assertEqual(3, len(dbm.Search("editbin", "00000100", 3)))
with self.assertRaises(StatusException):
self.assertRaises(dbm.Search("foo", "00000100", 3))
self.assertEqual(Status.SUCCESS, dbm.Close())
# Export tests.
def testExport(self):
dbm = DBM()
dest_path = self._make_tmp_path("casket.txt")
self.assertEqual(Status.SUCCESS, dbm.Open("", True))
for i in range(1, 101):
key = "{:08d}".format(i)
value = "{:d}".format(i)
self.assertEqual(Status.SUCCESS, dbm.Set(key, value, False))
file = File()
self.assertEqual(Status.SUCCESS, file.Open(dest_path, True, truncate=True))
self.assertEqual(Status.SUCCESS, dbm.ExportToFlatRecords(file))
self.assertEqual(Status.SUCCESS, dbm.Clear())
self.assertEqual(0, dbm.Count())
self.assertEqual(Status.SUCCESS, dbm.ImportFromFlatRecords(file))
self.assertEqual(100, dbm.Count())
self.assertEqual(Status.SUCCESS, file.Close())
file = File()
self.assertEqual(Status.SUCCESS, file.Open(dest_path, True, truncate=True))
self.assertEqual(Status.SUCCESS, dbm.ExportKeysAsLines(file))
self.assertEqual(Status.SUCCESS, file.Close())
self.assertEqual(Status.SUCCESS, dbm.Close())
file = File()
self.assertEqual(Status.SUCCESS, file.Open(dest_path, False))
self.assertTrue("File" in repr(file))
self.assertTrue("File" in str(file))
self.assertEqual(12, len(file.Search("contain", "001")))
self.assertEqual(3, len(file.Search("contain", "001", 3)))
self.assertEqual(10, len(file.Search("begin", "0000001")))
self.assertEqual(10, len(file.Search("end", "1")))
self.assertEqual(10, len(file.Search("regex", r"^\d+1$")))
self.assertEqual(10, len(file.Search("regex", r"^\d+1$", 0)))
self.assertEqual(3, len(file.Search("edit", "00000100", 3)))
self.assertEqual(3, len(file.Search("editbin", "00000100", 3)))
with self.assertRaises(StatusException):
self.assertRaises(file.Search("foo", "00000100", 3))
self.assertEqual(Status.SUCCESS, file.Close())
# AsyncDBM tests.
def testAsyncDBM(self):
dbm = DBM()
path = self._make_tmp_path("casket.tkh")
copy_path = self._make_tmp_path("casket-copy.tkh")
self.assertEqual(Status.SUCCESS, dbm.Open(path, True, num_buckets=100, concurrent=True))
adbm = AsyncDBM(dbm, 4)
self.assertTrue("AsyncDBM" in repr(adbm))
self.assertTrue("AsyncDBM" in str(adbm))
set_future = adbm.Set("one", "hop")
self.assertTrue("Future" in repr(set_future))
self.assertTrue("Future" in str(set_future))
set_future.Wait(0)
self.assertTrue(set_future.Wait())
self.assertEqual(Status.SUCCESS, set_future.Get())
self.assertEqual(Status.DUPLICATION_ERROR, adbm.Set("one", "more", False).Get())
self.assertEqual(Status.SUCCESS, adbm.Set("two", "step", False).Get())
self.assertEqual(Status.SUCCESS, adbm.Set("three", "jump", False).Get())
self.assertEqual(Status.SUCCESS, adbm.Append("three", "go", ":").Get())
get_result = adbm.Get("one").Get()
self.assertEqual(Status.SUCCESS, get_result[0])
self.assertEqual(b"hop", get_result[1])
self.assertEqual("step", adbm.GetStr("two").Get()[1])
self.assertEqual("jump:go", adbm.GetStr("three").Get()[1])
self.assertEqual(Status.SUCCESS, adbm.Get("three").Get()[0])
self.assertEqual(Status.SUCCESS, adbm.Remove("one").Get())
self.assertEqual(Status.NOT_FOUND_ERROR, adbm.Remove("one").Get())
self.assertEqual(Status.SUCCESS, adbm.Remove("two").Get())
self.assertEqual(Status.SUCCESS, adbm.Remove("three").Get())
self.assertEqual(0, dbm.Count())
set_future = adbm.SetMulti(False, one="hop", two="step", three="jump")
self.assertEqual(Status.SUCCESS, set_future.Get())
self.assertEqual(Status.SUCCESS, adbm.AppendMulti(":", three="go").Get())
get_result = adbm.GetMulti("one", "two").Get()
self.assertEqual(Status.SUCCESS, get_result[0])
self.assertEqual(b"hop", get_result[1][b"one"])
self.assertEqual(b"step", get_result[1][b"two"])
get_result = adbm.GetMultiStr("one", "two", "three", "hoge").Get()
self.assertEqual(Status.NOT_FOUND_ERROR, get_result[0])
self.assertEqual("hop", get_result[1]["one"])
self.assertEqual("step", get_result[1]["two"])
self.assertEqual("jump:go", get_result[1]["three"])
self.assertEqual(Status.SUCCESS, adbm.RemoveMulti("one", "two", "three").Get())
self.assertEqual(0, dbm.Count())
incr_result = adbm.Increment("num", 5, 100).Get()
self.assertEqual(Status.SUCCESS, incr_result[0])
self.assertEqual(105, incr_result[1])
self.assertEqual(110, adbm.Increment("num", 5, 100).Get()[1])
self.assertEqual(Status.SUCCESS, adbm.Remove("num").Get())
self.assertEqual(Status.SUCCESS, adbm.CompareExchange("one", None, "ichi").Get())
self.assertEqual("ichi", adbm.GetStr("one").Get()[1])
self.assertEqual(Status.SUCCESS, adbm.CompareExchange("one", "ichi", "ni").Get())
self.assertEqual("ni", adbm.GetStr("one").Get()[1])
self.assertEqual(Status.INFEASIBLE_ERROR, adbm.CompareExchange(
"xyz", DBM.ANY_DATA, DBM.ANY_DATA).Get())
self.assertEqual(Status.SUCCESS, adbm.CompareExchange(
"xyz", None, "abc").Get())
self.assertEqual(Status.SUCCESS, adbm.CompareExchange(
"xyz", DBM.ANY_DATA, DBM.ANY_DATA).Get())
self.assertEqual("abc", adbm.GetStr("xyz").Get()[1])
self.assertEqual(Status.SUCCESS, adbm.CompareExchange("xyz", DBM.ANY_DATA, "def").Get())
self.assertEqual("def", adbm.GetStr("xyz").Get()[1])
self.assertEqual(Status.SUCCESS, adbm.CompareExchange("xyz", DBM.ANY_DATA, None).Get())
self.assertEqual(Status.NOT_FOUND_ERROR, adbm.GetStr("xyz").Get()[0])
self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti(
[("one", "ni"), ("two", None)], [("one", "san"), ("two", "uno")]).Get())
self.assertEqual("san", adbm.GetStr("one").Get()[1])
self.assertEqual("uno", adbm.GetStr("two").Get()[1])
self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti(
[("one", "san"), ("two", "uno")], [("one", None), ("two", None)]).Get())
self.assertEqual(Status.INFEASIBLE_ERROR, adbm.CompareExchangeMulti(
[("xyz", DBM.ANY_DATA)], [("xyz", "abc")]).Get())
self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti(
[("xyz", None)], [("xyz", "abc")]).Get())
self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti(
[("xyz", DBM.ANY_DATA)], [("xyz", "abc")]).Get())
self.assertEqual("abc", adbm.GetStr("xyz").Get()[1])
self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti(
[("xyz", DBM.ANY_DATA)], [("xyz", None)]).Get())
self.assertEqual(Status.NOT_FOUND_ERROR, adbm.GetStr("xyz").Get()[0])
self.assertEqual(0, dbm.Count())
self.assertEqual(Status.SUCCESS, adbm.Set("hello", "world", False).Get())
self.assertEqual(Status.SUCCESS, adbm.Synchronize(False).Get())
self.assertEqual(Status.SUCCESS, adbm.Rebuild().Get())
self.assertEqual(1, dbm.Count())
self.assertEqual(Status.SUCCESS, adbm.CopyFileData(copy_path).Get())
copy_dbm = DBM()
self.assertEqual(Status.SUCCESS, copy_dbm.Open(copy_path, True))
self.assertEqual(1, copy_dbm.Count())
self.assertEqual(Status.SUCCESS, copy_dbm.Clear())
self.assertEqual(0, copy_dbm.Count())
self.assertEqual(Status.SUCCESS, adbm.Export(copy_dbm).Get())
self.assertEqual(1, copy_dbm.Count())
self.assertEqual(Status.SUCCESS, copy_dbm.Close())
os.remove(copy_path)
copy_file = File()
self.assertEqual(Status.SUCCESS, copy_file.Open(copy_path, True))
self.assertEqual(Status.SUCCESS, adbm.ExportToFlatRecords(copy_file).Get())
self.assertEqual(Status.SUCCESS, adbm.Clear().Get())
self.assertEqual(0, dbm.Count())
self.assertEqual(Status.SUCCESS, adbm.ImportFromFlatRecords(copy_file).Get())
self.assertEqual(1, dbm.Count())
self.assertEqual(Status.SUCCESS, copy_file.Close())
async def async_main():
await adbm.Set("hello", "good-bye", True)
await adbm.Set("hi", "bye", True)
await adbm.Set("chao", "adios", True)
asyncio.run(async_main())
self.assertEqual("good-bye", dbm.GetStr("hello"))
search_result = adbm.Search("begin", "h").Get()
self.assertEqual(Status.SUCCESS, search_result[0])
self.assertEqual(2, len(search_result[1]))
self.assertTrue("hello" in search_result[1])
self.assertTrue("hi" in search_result[1])
self.assertEqual(Status.SUCCESS, adbm.Clear().Get())
self.assertEqual(Status.SUCCESS, adbm.Set("aa", "AAA").Get())
self.assertEqual(Status.SUCCESS, adbm.Rekey("aa", "bb").Get())
get_result = adbm.GetStr("bb").Get()
self.assertEqual(Status.SUCCESS, get_result[0])
self.assertEqual("AAA", get_result[1])
pop_result = adbm.PopFirst().Get()
self.assertEqual(Status.SUCCESS, pop_result[0])
self.assertEqual(b"bb", pop_result[1])
self.assertEqual(b"AAA", pop_result[2])
self.assertEqual(Status.SUCCESS, adbm.Set("cc", "CCC").Get())
pop_result = adbm.PopFirstStr().Get()
self.assertEqual(Status.SUCCESS, pop_result[0])
self.assertEqual("cc", pop_result[1])
self.assertEqual("CCC", pop_result[2])
self.assertEqual(Status.SUCCESS, adbm.PushLast("foo", 0).Get())
pop_result = adbm.PopFirst().Get()
self.assertEqual(Status.SUCCESS, pop_result[0])
self.assertEqual(b"\0\0\0\0\0\0\0\0", pop_result[1])
self.assertEqual(b"foo", pop_result[2])
adbm.Destruct()
self.assertEqual(Status.SUCCESS, dbm.Close())
# File tests.
def testFile(self):
path = self._make_tmp_path("casket.txt")
file = File()
self.assertEqual(Status.SUCCESS, file.Open(
path, True, truncate=True, file="pos-atom", block_size=512,
access_options="padding:pagecache"))
self.assertEqual(Status.SUCCESS, file.Write(5, "12345"))
self.assertEqual(Status.SUCCESS, file.Write(0, "ABCDE"))
self.assertEqual(10, file.Append("FGH"))
self.assertEqual(13, file.Append("IJ"))
self.assertEqual(15, file.GetSize())
self.assertTrue(file.GetPath().find("casket.txt") > 0)
self.assertEqual(Status.SUCCESS, file.Synchronize(False))
self.assertEqual(Status.SUCCESS, file.Truncate(12))
self.assertEqual(12, file.GetSize())
self.assertEqual(b"ABCDE12345FG", file.Read(0, 12))
self.assertEqual("ABCDE12345FG", file.ReadStr(0, 12))
self.assertEqual(b"DE123", file.Read(3, 5))
self.assertEqual("DE123", file.ReadStr(3, 5))
status = Status()
self.assertEqual(None, file.ReadStr(1024, 10, status))
self.assertEqual(Status.INFEASIBLE_ERROR, status)
self.assertEqual(Status.SUCCESS, file.Close())
self.assertEqual(Status.SUCCESS, file.Open(path, False))
self.assertEqual(512, file.GetSize())
self.assertEqual("E12345F", file.ReadStr(4, 7))
self.assertEqual(Status.SUCCESS, file.Close())
# Main routine.
def main(argv):
test_names = argv
if test_names:
test_suite = unittest.TestSuite()
for test_name in test_names:
test_suite.addTest(TestTkrzw(test_name))
else:
test_suite = unittest.TestLoader().loadTestsFromTestCase(TestTkrzw)
unittest.TextTestRunner(verbosity=2).run(test_suite)
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
# END OF FILE
tkrzw-python-0.1.30/example3.py 0000775 0001750 0001750 00000003667 14402705440 015406 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Example for the asynchronous API
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import asyncio
import tkrzw
async def main():
# Prepares the database.
dbm = tkrzw.DBM()
dbm.Open("casket.tkh", True, truncate=True, num_buckets=100)
# Prepares the asynchronous adapter with 4 worker threads.
adbm = tkrzw.AsyncDBM(dbm, 4)
# Executes the Set method asynchronously.
future = adbm.Set("hello", "world")
# Does something in the foreground.
print("Setting a record")
# Checks the result after awaiting the Set operation.
# Calling Future#Get doesn't yield the coroutine ownership.
status = future.Get()
if status != tkrzw.Status.SUCCESS:
print("ERROR: " + str(status))
# Executes the Get method asynchronously.
future = adbm.GetStr("hello")
# Does something in the foreground.
print("Getting a record")
# Awaits the operation while yielding the coroutine ownership.
await future
status, value = future.Get()
if status == tkrzw.Status.SUCCESS:
print("VALUE: " + value)
# Releases the asynchronous adapter.
adbm.Destruct()
# Closes the database.
dbm.Close()
asyncio.run(main())
# END OF FILE
tkrzw-python-0.1.30/wicked.py 0000775 0001750 0001750 00000011552 14402705440 015126 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Wicked test cases
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import argparse
import sys
import os
import re
import random
import time
import threading
import shutil
from tkrzw import *
# main routine
def main(argv):
ap = argparse.ArgumentParser(
prog="wicked.py", description="Random operation tests.",
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--path", default="")
ap.add_argument("--params", default="")
ap.add_argument("--iter", type=int, default=10000)
ap.add_argument("--threads", type=int, default=1)
ap.add_argument("--random", action='store_true', default=False)
args = ap.parse_args(argv)
path = args.path
open_params = {}
open_params_exprs = []
for field in args.params.split(","):
columns = field.split("=", 1)
if len(columns) == 2:
open_params[columns[0]] = columns[1]
open_params_exprs.append(columns[0] + "=" + columns[1])
num_iterations = args.iter
num_threads = args.threads
print("path: {}".format(path))
print("params: {}".format(",".join(open_params_exprs)))
print("num_iterations: {}".format(num_iterations))
print("num_threads: {}".format(num_threads))
print("")
open_params["truncate"] = True
start_mem_usage = Utility.GetMemoryUsage()
dbm = DBM()
dbm.Open(path, True, **open_params).OrDie()
class Task(threading.Thread):
def __init__(self, thid):
threading.Thread.__init__(self)
self.thid = thid
def run(self):
rnd_state = random.Random()
for i in range(0, num_iterations):
key_num = rnd_state.randint(1, num_iterations)
key = "{:d}".format(key_num)
value = "{:d}".format(i)
if rnd_state.randint(0, num_iterations / 2) == 0:
dbm.Rebuild().OrDie()
elif rnd_state.randint(0, num_iterations / 2) == 0:
dbm.Clear().OrDie()
elif rnd_state.randint(0, num_iterations / 2) == 0:
dbm.Synchronize(False).OrDie()
elif rnd_state.randint(0, 100) == 0:
it = dbm.MakeIterator()
if dbm.IsOrdered() and rnd_state.randint(0, 3) == 0:
if rnd_state.randint(0, 3) == 0:
it.Jump(key)
else:
it.Last()
while rnd_state.randint(0, 10) == 0:
status = Status()
it.Get(status)
if status != Status.NOT_FOUND_ERROR:
status.OrDie()
it.Previous()
else:
if rnd_state.randint(0, 3) == 0:
it.Jump(key)
else:
it.First()
while rnd_state.randint(0, 10) == 0:
status = Status()
it.Get(status)
if status != Status.NOT_FOUND_ERROR:
status.OrDie()
it.Next()
elif rnd_state.randint(0, 3) == 0:
status = Status()
dbm.Get(key, status)
if status != Status.NOT_FOUND_ERROR:
status.OrDie()
elif rnd_state.randint(0, 3) == 0:
status = dbm.Remove(key)
if status != Status.NOT_FOUND_ERROR:
status.OrDie()
elif rnd_state.randint(0, 3) == 0:
status = dbm.Set(key, value, False)
if status != Status.DUPLICATION_ERROR:
status.OrDie()
else:
dbm.Set(key, value).OrDie()
seq = i + 1
if self.thid == 0 and seq % (num_iterations / 500) == 0:
print(".", end="")
if seq % (num_iterations / 10) == 0:
print(" ({:08d})".format(seq))
sys.stdout.flush()
print("Doing:")
start_time = time.time()
threads = []
for thid in range(0, num_threads):
th = Task(thid)
th.start()
threads.append(th)
for th in threads:
th.join()
dbm.Synchronize(False).OrDie()
end_time = time.time()
elapsed = end_time - start_time
mem_usage = Utility.GetMemoryUsage() - start_mem_usage
print("Done: num_records={:d} file_size={:d} time={:.3f} qps={:.0f} mem={:d}".format(
dbm.Count(), dbm.GetFileSize() or -1,
elapsed, num_iterations * num_threads / elapsed, mem_usage))
print("")
dbm.Close().OrDie()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
# END OF FILE
tkrzw-python-0.1.30/coro.py 0000775 0001750 0001750 00000006266 14402705440 014630 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Coroutine tests
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import asyncio
import time
import tkrzw
NUM_ITERS = 100
NUM_COROUTINES = 1000
NUM_REPEATS = 100
async def set_sync(dbm, cor_id, iter_count):
for i in range(0, NUM_REPEATS):
key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i)
dbm.Set(key, key).OrDie()
async def set_async(adbm, cor_id, iter_count):
futures = []
for i in range(0, NUM_REPEATS):
key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i)
futures.append(adbm.Set(key, key))
for future in futures:
future.Get().OrDie()
async def get_sync(dbm, cor_id, iter_count):
for i in range(0, NUM_REPEATS):
key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i)
status = tkrzw.Status()
dbm.Get(key, status)
status.OrDie()
async def get_async(adbm, cor_id, iter_count):
futures = []
for i in range(0, NUM_REPEATS):
key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i)
futures.append(adbm.Get(key))
for future in futures:
status, value = future.Get()
status.OrDie()
async def main():
dbm = tkrzw.DBM()
num_buckets = NUM_ITERS * NUM_COROUTINES * NUM_REPEATS / 2
dbm.Open("casket.tkh", True, concurrent=True, truncate=True,
num_buckets=num_buckets, file="pos-para")
adbm = tkrzw.AsyncDBM(dbm, 4)
def make_set_sync(cor_id, iter_count):
return set_sync(dbm, cor_id, iter_count)
def make_set_async(cor_id, iter_count):
return set_async(adbm, cor_id, iter_count)
def make_get_sync(cor_id, iter_count):
return get_sync(dbm, cor_id, iter_count)
def make_get_async(cor_id, iter_count):
return get_async(adbm, cor_id, iter_count)
confs = [
{"label": "SET SYNC", "op": make_set_sync},
{"label": "SET ASYNC", "op": make_set_async},
{"label": "GET SYNC", "op": make_get_sync},
{"label": "GET ASYNC", "op": make_get_async},
]
for conf in confs:
start_time = time.time()
for iter_count in range(0, NUM_ITERS):
coroutines = []
for cor_id in range(0, NUM_COROUTINES):
coroutines.append(conf["op"](cor_id, iter_count))
for coroutine in coroutines:
await coroutine
end_time = time.time()
elapsed = end_time - start_time
print("{:10s}: {:8.0f} QPS".format(
conf["label"], (NUM_ITERS * NUM_COROUTINES * NUM_REPEATS) / elapsed))
adbm.Destruct()
dbm.Close()
asyncio.run(main())
# END OF FILE
tkrzw-python-0.1.30/CONTRIBUTING.md 0000664 0001750 0001750 00000002111 14402705440 015523 0 ustar mikio mikio # How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google/conduct/).
tkrzw-python-0.1.30/Makefile 0000664 0001750 0001750 00000010310 14402705440 014732 0 ustar mikio mikio # Makefile for Tkrzw for Python
PACKAGE = tkrzw-python
VERSION = 0.1.30
PACKAGEDIR = $(PACKAGE)-$(VERSION)
PACKAGETGZ = $(PACKAGE)-$(VERSION).tar.gz
PYTHON = python3
RUNENV = LD_LIBRARY_PATH=.:/lib:/usr/lib:/usr/local/lib:$(HOME)/local/lib:$(HOME)/lib:$(LD_LIBRARY_PATH)
all :
$(PYTHON) setup.py build
cp -f build/*/*.so .
@printf '\n'
@printf '#================================================================\n'
@printf '# Ready to install.\n'
@printf '#================================================================\n'
clean :
rm -rf casket casket* *~ *.tmp *.tkh *.tkt *.tks *.flat *.log *.so *.pyc build \
hoge moge tako ika uni
install :
$(PYTHON) setup.py install
-chown -R `stat -c "%U:%G" Makefile` dist *.egg-info
@printf '\n'
@printf '#================================================================\n'
@printf '# Thanks for using Tkrzw for Python.\n'
@printf '#================================================================\n'
uninstall :
$(PYTHON) setup.py install --record files.tmp
xargs rm -f < files.tmp
dist :
$(MAKE) distclean
rm -Rf "../$(PACKAGEDIR)" "../$(PACKAGETGZ)"
cd .. && cp -R tkrzw-python $(PACKAGEDIR) && \
tar --exclude=".*" -cvf - $(PACKAGEDIR) | gzip -c > $(PACKAGETGZ)
rm -Rf "../$(PACKAGEDIR)"
sync ; sync
distclean : clean apidocclean
rm -rf dist *.egg-info
check :
$(RUNENV) $(PYTHON) test.py
$(RUNENV) $(PYTHON) perf.py --path casket.tkh --params "num_buckets=100000" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) perf.py --path casket.tkh --params "concurrent=true,num_buckets=100000" \
--iter 20000 --threads 5 --random
$(RUNENV) $(PYTHON) perf.py --path casket.tkt --params "key_comparator=decimal" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) perf.py --path casket.tkt --params "concurrent=true,key_comparator=decimal" \
--iter 20000 --threads 5 --random
$(RUNENV) $(PYTHON) perf.py --path casket.tks --params "step_unit=3" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) perf.py --path casket.tks --params "concurrent=true,step_unit=3" \
--iter 20000 --threads 5 --random
$(RUNENV) $(PYTHON) perf.py --params "dbm=tiny,num_buckets=100000" \
--iter 20000 --threads 5 --random
$(RUNENV) $(PYTHON) perf.py --params "dbm=baby,key_comparator=decimal" \
--iter 20000 --threads 5 --random
$(RUNENV) $(PYTHON) perf.py --params "dbm=stdhash,num_buckets=100000" \
--iter 20000 --threads 5 --async_threads 2 --random
$(RUNENV) $(PYTHON) perf.py --params "dbm=stdtree" \
--iter 20000 --threads 5 --async_threads 2 --random
$(RUNENV) $(PYTHON) wicked.py --path casket.tkh --params "num_buckets=100000" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) wicked.py --path casket.tkt --params "key_comparator=decimal" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) wicked.py --path casket.tks --params "step_unit=3" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) wicked.py --params "dbm=tiny,num_buckets=100000" \
--iter 20000 --threads 5
$(RUNENV) $(PYTHON) wicked.py --params "dbm=baby,key_comparator=decimal" \
--iter 20000 --threads 5
@printf '\n'
@printf '#================================================================\n'
@printf '# Checking completed.\n'
@printf '#================================================================\n'
apidoc :
$(MAKE) apidocclean
mkdir -p tmp-doc
cp tkrzw-doc.py tmp-doc/tkrzw.py
cd tmp-doc ; sphinx-apidoc -F -H Tkrzw -A "Mikio Hirabayashi" -o out .
cat tmp-doc/out/conf.py |\
sed -e 's/^# import /import /' -e 's/^# sys.path/sys.path/' \
-e 's/alabaster/haiku/' \
-e '/sphinx\.ext\.viewcode/d' \
-e '/^extensions = /a "sphinx.ext.autosummary",' > tmp-doc/out/conf.py.tmp
echo >> tmp-doc/out/conf.py.tmp
echo "autodoc_member_order = 'bysource'" >> tmp-doc/out/conf.py.tmp
echo "html_title = 'Python binding of Tkrzw'" >> tmp-doc/out/conf.py.tmp
echo "autodoc_default_options = {'members': True, 'special-members': True, 'exclude-members': '__dict__,__module__,__weakref__'}" >> tmp-doc/out/conf.py.tmp
mv -f tmp-doc/out/conf.py.tmp tmp-doc/out/conf.py
cp -f index.rst tmp-doc/out/index.rst
cd tmp-doc/out ; $(MAKE) html
mv tmp-doc/out/_build/html api-doc
apidocclean :
rm -rf api-doc tmp-doc
.PHONY: all clean install uninstall dist distclean check apidoc apidocclean
# END OF FILE
tkrzw-python-0.1.30/example1.py 0000775 0001750 0001750 00000003300 14402705440 015364 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# Example for basic usage of the hash database
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
import tkrzw
# Prepares the database.
dbm = tkrzw.DBM()
dbm.Open("casket.tkh", True, truncate=True, num_buckets=100)
# Sets records.
# If the operation fails, a runtime exception is raised.
# Keys and values are implicitly converted into bytes.
dbm["first"] = "hop"
dbm["second"] = "step"
dbm["third"] = "jump"
# Retrieves record values.
# If the operation fails, a runtime exception is raised.
# Retrieved values are strings if keys are strings.
print(dbm["first"])
print(dbm["second"])
print(dbm["third"])
try:
print(dbm["fourth"])
except tkrzw.StatusException as e:
print(repr(e))
# Checks and deletes a record.
if "first" in dbm:
del dbm["first"]
# Traverses records.
# Retrieved keys and values are always bytes so we decode them.
for key, value in dbm:
print(key.decode(), value.decode())
# Closes the database.
dbm.Close()
# END OF FILE
tkrzw-python-0.1.30/COPYING 0000664 0001750 0001750 00000026136 14402705440 014342 0 ustar mikio mikio
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
tkrzw-python-0.1.30/ChangeLog 0000664 0001750 0001750 00000007206 14402705440 015056 0 ustar mikio mikio 2023-03-11 Mikio Hirabayashi
- Release: 0.1.30
- Options to enable encrypted databases are added.
- Compatibility to the C++ library: Tkrzw 1.0.27
2023-02-22 Mikio Hirabayashi
- Release: 0.1.29
- Process, ProcessMulti, and ProcessEach are added to DBM.
2021-11-13 Mikio Hirabayashi
- Release: 0.1.28
- Build setting is modified for separate atomic libraries.
- One database object can be opened/closed multiple times.
- Add __contains__ method for "in" operator.
2021-11-02 Mikio Hirabayashi
- Release: 0.1.27
- DBM::CompareExchangeAndGet is added.
- Compatibility to the C++ library: Tkrzw 1.0.20
2021-10-30 Mikio Hirabayashi
- Release: 0.1.26
- DBM::CompareExchange supports ANY_DATA.
- Compatibility to the C++ library: Tkrzw 1.0.19
2021-10-12 Mikio Hirabayashi
- Release: 0.1.25
- Add DBM::Rekey.
- Add Iterator::Step.
- Add DBM::PopFirst and DBM::PushLast.
- Conversion from bytes to Unicode uses "replace" mode.
- Compatibility to the C++ library: Tkrzw 1.0.18
2021-10-10 Mikio Hirabayashi
- Release: 0.1.24
- lock_mem_buckets is ablished.
- Compatibility to the C++ library: Tkrzw 1.0.17
- Add Status.CodeName.
2021-09-20 Mikio Hirabayashi
- Release: 0.1.23
- sync_hard of the open options is supported.
- Compatibility to the C++ library: Tkrzw 1.0.12
2021-09-19 Mikio Hirabayashi
- Release: 0.1.22
- Add GetTimestamp method.
- Compatibility to the C++ library: Tkrzw 1.0.11
2021-08-27 Mikio Hirabayashi
- Release: 0.1.21
- Compatibility to the C++ library: Tkrzw 1.0.7
2021-08-21 Mikio Hirabayashi
- Release: 0.1.20
- Compatibility to the C++ library: Tkrzw 1.0.2
2021-08-02 Mikio Hirabayashi
- Release: 0.1.19
- Future#Get releases the resource.
2021-07-29 Mikio Hirabayashi
- Release: 0.1.18
- Add Join method to Status.
- Add AsyncDBM class.
2021-07-28 Mikio Hirabayashi
- Release: 0.1.17
- Modify export methods.
- Compatibility to the C++ library: Tkrzw 0.9.48
2021-07-25 Mikio Hirabayashi
- Release: 0.1.16
- Add PAGE_SIZE constant.
- Add File.GetPath method.
- Compatibility to the C++ library: Tkrzw 0.9.47
2021-07-23 Mikio Hirabayashi
- Release: 0.1.15
- Add IsWritable method.
2021-07-14 Mikio Hirabayashi
- Release: 0.1.14
- Add AppendMulti method.
- Compatibility to the C++ library: Tkrzw 0.9.43
2021-07-12 Mikio Hirabayashi
- Release: 0.1.13
- Add exporting methods to the flat record file.
- Compatibility to the C++ library: Tkrzw 0.9.42
2021-07-08 Mikio Hirabayashi
- Release: 0.1.12
- Modify DBM::Search for the latest C++ API.
- Compatibility to the C++ library: Tkrzw 0.9.40
2021-07-05 Mikio Hirabayashi
- Release: 0.1.11
- The RestoreDatabase method is added.
- TextFile is renamed to File.
- Compatibility to the C++ library: Tkrzw 0.9.38
2021-06-02 Mikio Hirabayashi
- Release: 0.1.10
- Add CompareExchangeMulti method.
2021-05-29 Mikio Hirabayashi
- Release: 0.1.9
- Add RemoveMulti method.
- CompareExchange supports expectation for missing records.
- Compatibility to the C++ library: Tkrzw 0.9.21
2021-05-16 Mikio Hirabayashi
- Release: 0.1.8
- Supports tunning parameters of Direct I/O.
- Compatibility to the C++ library: Tkrzw 0.9.17
tkrzw-python-0.1.30/tkrzw-doc.py 0000664 0001750 0001750 00000164651 14402705440 015612 0 ustar mikio mikio #! /usr/bin/python3
# -*- coding: utf-8 -*-
#--------------------------------------------------------------------------------------------------
# API document
#
# Copyright 2020 Google LLC
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
# except in compliance with the License. You may obtain a copy of the License at
# https://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#--------------------------------------------------------------------------------------------------
class Utility:
"""
Library utilities.
"""
VERSION = "0.0.0"
"""The package version numbers."""
OS_NAME = "unknown"
"""The recognized OS name."""
PAGE_SIZE = 4096
"""The size of a memory page on the OS."""
INT32MIN = -2 ** 31
"""The minimum value of int32."""
INT32MAX = 2 ** 31 - 1
"""The maximum value of int32."""
UINT32MAX = 2 ** 32 - 1
"""The maximum value of uint32."""
INT64MIN = -2 ** 63
"""The minimum value of int64."""
INT64MAX = 2 ** 63 - 1
"""The maximum value of int64."""
UINT64MAX = 2 ** 64 - 1
"""The maximum value of uint64."""
@classmethod
def GetMemoryCapacity(cls):
"""
Gets the memory capacity of the platform.
:return: The memory capacity of the platform in bytes, or -1 on failure.
"""
pass # native code
@classmethod
def GetMemoryUsage(cls):
"""
Gets the current memory usage of the process.
:return: The current memory usage of the process in bytes, or -1 on failure.
"""
pass # native code
@classmethod
def PrimaryHash(cls, data, num_buckets=None):
"""
Primary hash function for the hash database.
:param data: The data to calculate the hash value for.
:param num_buckets: The number of buckets of the hash table. If it is omitted, UINT64MAX is set.
:return: The hash value.
"""
pass # native code
@classmethod
def SecondaryHash(cls, data, num_shards=None):
"""
Secondary hash function for sharding.
:param data: The data to calculate the hash value for.
:param num_shards: The number of shards. If it is omitted, UINT64MAX is set.
:return: The hash value.
"""
pass # native code
@classmethod
def EditDistanceLev(cls, a, b):
"""
Gets the Levenshtein edit distance of two Unicode strings.
:param a: A Unicode string.
:param b: The other Unicode string.
:return: The Levenshtein edit distance of the two strings.
"""
pass # native code
class Status:
"""
Status of operations.
"""
SUCCESS = 0
"""Success."""
UNKNOWN_ERROR = 1
"""Generic error whose cause is unknown."""
SYSTEM_ERROR = 2
"""Generic error from underlying systems."""
NOT_IMPLEMENTED_ERROR = 3
"""Error that the feature is not implemented."""
PRECONDITION_ERROR = 4
"""Error that a precondition is not met."""
INVALID_ARGUMENT_ERROR = 5
"""Error that a given argument is invalid."""
CANCELED_ERROR = 6
"""Error that the operation is canceled."""
NOT_FOUND_ERROR = 7
"""Error that a specific resource is not found."""
PERMISSION_ERROR = 8
"""Error that the operation is not permitted."""
INFEASIBLE_ERROR = 9
"""Error that the operation is infeasible."""
DUPLICATION_ERROR = 10
"""Error that a specific resource is duplicated."""
BROKEN_DATA_ERROR = 11
"""Error that internal data are broken."""
NETWORK_ERROR = 12
"""Error caused by networking failure."""
APPLICATION_ERROR = 13
"""Generic error caused by the application logic."""
def __init__(self, code=SUCCESS, message=""):
"""
Sets the code and the message.
:param code: The status code. This can be omitted and then SUCCESS is set.
:param message: An arbitrary status message. This can be omitted and the an empty string is set.
"""
pass # native code
def __repr__(self):
"""
Returns a string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns a string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def __eq__(self, rhs):
"""
Returns true if the given object is equivalent to this object.
:return: True if the given object is equivalent to this object.
This supports comparison between a status object and a status code number.
"""
pass # native code
def Set(self, code=SUCCESS, message=""):
"""
Sets the code and the message.
:param code: The status code. This can be omitted and then SUCCESS is set.
:param message: An arbitrary status message. This can be omitted and the an empty string is set.
"""
pass # native code
def Join(self, rht):
"""
Assigns the internal state from another status object only if the current state is success.
:param rhs: The status object.
"""
pass # native code
def GetCode(self):
"""
Gets the status code.
:return: The status code.
"""
pass # native code
def GetMessage(self):
"""
Gets the status message.
:return: The status message.
"""
pass # native code
def IsOK(self):
"""
Returns true if the status is success.
:return: True if the status is success, or False on failure.
"""
pass # native code
def OrDie(self):
"""
Raises an exception if the status is not success.
:raise StatusException: An exception containing the status object.
"""
pass # native code
@classmethod
def CodeName(cls, code):
"""
Gets the string name of a status code.
:param: code The status code.
:return: The name of the status code.
"""
pass # native code
class Future:
"""
Future containing a status object and extra data.
Future objects are made by methods of AsyncDBM. Every future object should be destroyed by the "Destruct" method or the "Get" method to free resources. This class implements the awaitable protocol so an instance is usable with the "await" sentence.
"""
def __init__(self):
"""
The constructor cannot be called directly. Use methods of AsyncDBM.
"""
pass # native code
def __repr__(self):
"""
Returns A string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns a string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def __await__(self):
"""
Waits for the operation to be done and returns an iterator.
:return: The iterator which stops immediately.
"""
pass # native code
def Wait(self, timeout=-1):
"""
Waits for the operation to be done.
:param timeout: The waiting time in seconds. If it is negative, no timeout is set.
:return: True if the operation has done. False if timeout occurs.
"""
pass # native code
def Get(self):
"""
Waits for the operation to be done and gets the result status.
:return: The result status and extra data if any. The existence and the type of extra data depends on the operation which makes the future. For DBM#Get, a tuple of the status and the retrieved value is returned. For DBM#Set and DBM#Remove, the status object itself is returned.
The internal resource is released by this method. "Wait" and "Get" cannot be called after calling this method.
"""
pass # native code
class StatusException(RuntimeError):
"""
Exception to convey the status of operations.
"""
def __init__(self, status):
"""
Sets the status.
:param status: The status object.
"""
pass # native code
def __repr__(self):
"""
Returns A string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns A string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def GetStatus(self):
"""
Gets the status object
:return: The status object.
"""
pass # native code
class DBM:
"""
Polymorphic database manager.
All operations except for Open and Close are thread-safe; Multiple threads can access the same database concurrently. You can specify a data structure when you call the Open method. Every opened database must be closed explicitly by the Close method to avoid data corruption.
This class implements the iterable protocol so an instance is usable with "for-in" loop.
"""
ANY_DATA = b"\x00[ANY]\x00"
"""The special bytes value for no-operation or any data."""
def __init__(self):
"""
Does nothing especially.
"""
pass # native code
def __repr__(self):
"""
Returns A string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns A string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def __len__(self):
"""
Gets the number of records, to enable the len operator.
:return: The number of records on success, or 0 on failure.
"""
pass # native code
def __getitem__(self, key):
"""
Gets the value of a record, to enable the [] operator.
:param key: The key of the record.
:return: The value of the matching record. An exception is raised for missing records. If the given key is a string, the returned value is also a string. Otherwise, the return value is bytes.
:raise StatusException: An exception containing the status object.
"""
pass # native code
def __contains__(self, key):
"""
Checks if a record exists or not, to enable the in operator.
:param key: The key of the record.
:return: True if the record exists, or False if not. No exception is raised for missing records.
"""
pass # native code
def __setitem__(self, key, value):
"""
Sets a record of a key and a value, to enable the []= operator.
:param key: The key of the record.
:param value: The value of the record.
:raise StatusException: An exception containing the status object.
"""
pass # native code
def __delitem__(self, key):
"""
Removes a record of a key, to enable the del [] operator.
:param key: The key of the record.
:raise StatusException: An exception containing the status object.
"""
pass # native code
def __iter__(self):
"""
Makes an iterator and initialize it, to comply to the iterator protocol.
:return: The iterator for each record.
"""
pass # native code
def Open(self, path, writable, **params):
"""
Opens a database file.
:param path: A path of the file.
:param writable: If true, the file is writable. If false, it is read-only.
:param params: Optional keyword parameters.
:return: The result status.
The extension of the path indicates the type of the database.
- .tkh : File hash database (HashDBM)
- .tkt : File tree database (TreeDBM)
- .tks : File skip database (SkipDBM)
- .tkmt : On-memory hash database (TinyDBM)
- .tkmb : On-memory tree database (BabyDBM)
- .tkmc : On-memory cache database (CacheDBM)
- .tksh : On-memory STL hash database (StdHashDBM)
- .tkst : On-memory STL tree database (StdTreeDBM)
The optional parameters can include an option for the concurrency tuning. By default, database operatins are done under the GIL (Global Interpreter Lock), which means that database operations are not done concurrently even if you use multiple threads. If the "concurrent" parameter is true, database operations are done outside the GIL, which means that database operations can be done concurrently if you use multiple threads. However, the downside is that swapping thread data is costly so the actual throughput is often worse in the concurrent mode than in the normal mode. Therefore, the concurrent mode should be used only if the database is huge and it can cause blocking of threads in multi-thread usage.
The optional parameters can include options for the file opening operation.
- truncate (bool): True to truncate the file.
- no_create (bool): True to omit file creation.
- no_wait (bool): True to fail if the file is locked by another process.
- no_lock (bool): True to omit file locking.
- sync_hard (bool): True to do physical synchronization when closing.
The optional parameter "dbm" supercedes the decision of the database type by the extension. The value is the type name: "HashDBM", "TreeDBM", "SkipDBM", "TinyDBM", "BabyDBM", "CacheDBM", "StdHashDBM", "StdTreeDBM".
The optional parameter "file" specifies the internal file implementation class. The default file class is "MemoryMapAtomicFile". The other supported classes are "StdFile", "MemoryMapAtomicFile", "PositionalParallelFile", and "PositionalAtomicFile".
For HashDBM, these optional parameters are supported.
- update_mode (string): How to update the database file: "UPDATE_IN_PLACE" for the in-palce or "UPDATE_APPENDING" for the appending mode.
- record_crc_mode (string): How to add the CRC data to the record: "RECORD_CRC_NONE" to add no CRC to each record, "RECORD_CRC_8" to add CRC-8 to each record, "RECORD_CRC_16" to add CRC-16 to each record, or "RECORD_CRC_32" to add CRC-32 to each record.
- record_comp_mode (string): How to compress the record data: "RECORD_COMP_NONE" to do no compression, "RECORD_COMP_ZLIB" to compress with ZLib, "RECORD_COMP_ZSTD" to compress with ZStd, "RECORD_COMP_LZ4" to compress with LZ4, "RECORD_COMP_LZMA" to compress with LZMA, "RECORD_COMP_RC4" to cipher with RC4, "RECORD_COMP_AES" to cipher with AES.
- offset_width (int): The width to represent the offset of records.
- align_pow (int): The power to align records.
- num_buckets (int): The number of buckets for hashing.
- restore_mode (string): How to restore the database file: "RESTORE_SYNC" to restore to the last synchronized state, "RESTORE_READ_ONLY" to make the database read-only, or "RESTORE_NOOP" to do nothing. By default, as many records as possible are restored.
- fbp_capacity (int): The capacity of the free block pool.
- min_read_size (int): The minimum reading size to read a record.
- cache_buckets (bool): True to cache the hash buckets on memory.
- cipher_key (string): The encryption key for cipher compressors.
For TreeDBM, all optional parameters for HashDBM are available. In addition, these optional parameters are supported.
- max_page_size (int): The maximum size of a page.
- max_branches (int): The maximum number of branches each inner node can have.
- max_cached_pages (int): The maximum number of cached pages.
- page_update_mode (string): What to do when each page is updated: "PAGE_UPDATE_NONE" is to do no operation or "PAGE_UPDATE_WRITE" is to write immediately.
- key_comparator (string): The comparator of record keys: "LexicalKeyComparator" for the lexical order, "LexicalCaseKeyComparator" for the lexical order ignoring case, "DecimalKeyComparator" for the order of the decimal integer numeric expressions, "HexadecimalKeyComparator" for the order of the hexadecimal integer numeric expressions, "RealNumberKeyComparator" for the order of the decimal real number expressions.
For SkipDBM, these optional parameters are supported.
- offset_width (int): The width to represent the offset of records.
- step_unit (int): The step unit of the skip list.
- max_level (int): The maximum level of the skip list.
- restore_mode (string): How to restore the database file: "RESTORE_SYNC" to restore to the last synchronized state or "RESTORE_NOOP" to do nothing make the database read-only. By default, as many records as possible are restored.
- sort_mem_size (int): The memory size used for sorting to build the database in the at-random mode.
- insert_in_order (bool): If true, records are assumed to be inserted in ascending order of the key.
- max_cached_records (int): The maximum number of cached records.
For TinyDBM, these optional parameters are supported.
- num_buckets (int): The number of buckets for hashing.
For BabyDBM, these optional parameters are supported.
- key_comparator (string): The comparator of record keys. The same ones as TreeDBM.
For CacheDBM, these optional parameters are supported.
- cap_rec_num (int): The maximum number of records.
- cap_mem_size (int): The total memory size to use.
All databases support taking update logs into files. It is enabled by setting the prefix of update log files.
- ulog_prefix (str): The prefix of the update log files.
- ulog_max_file_size (num): The maximum file size of each update log file. By default, it is 1GiB.
- ulog_server_id (num): The server ID attached to each log. By default, it is 0.
- ulog_dbm_index (num): The DBM index attached to each log. By default, it is 0.
For the file "PositionalParallelFile" and "PositionalAtomicFile", these optional parameters are supported.
- block_size (int): The block size to which all blocks should be aligned.
- access_options (str): Values separated by colon. "direct" for direct I/O. "sync" for synchrnizing I/O, "padding" for file size alignment by padding, "pagecache" for the mini page cache in the process.
If the optional parameter "num_shards" is set, the database is sharded into multiple shard files. Each file has a suffix like "-00003-of-00015". If the value is 0, the number of shards is set by patterns of the existing files, or 1 if they doesn't exist.
"""
pass # native code
def Close(self):
"""
Closes the database file.
:return: The result status.
"""
pass # native code
def Process(self, key, func, writable):
"""
Processes a record with an arbitrary function.
:param key: The key of the record.
:param func: The function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or None if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is None, the record is not modified. If the return value is False (not a false value but the False object), the record is removed.
:param writable: True if the processor can edit the record.
:return: The result status.
This method is not available in the concurrent mode because the function cannot be invoked outside the GIL.
"""
pass # native code
def Get(self, key, status=None):
"""
Gets the value of a record of a key.
:param key: The key of the record.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The bytes value of the matching record or None on failure.
"""
pass # native code
def GetStr(self, key, status=None):
"""
Gets the value of a record of a key, as a string.
:param key: The key of the record.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The string value of the matching record or None on failure.
"""
pass # native code
def GetMulti(self, *keys):
"""
Gets the values of multiple records of keys.
:param keys: The keys of records to retrieve.
:return: A map of retrieved records. Keys which don't match existing records are ignored.
"""
pass # native code
def GetMultiStr(self, *keys):
"""
Gets the values of multiple records of keys, as strings.
:param keys: The keys of records to retrieve.
:return: A map of retrieved records. Keys which don't match existing records are ignored.
"""
pass # native code
def Set(self, key, value, overwrite=True):
"""
Sets a record of a key and a value.
:param key: The key of the record.
:param value: The value of the record.
:param overwrite: Whether to overwrite the existing value. It can be omitted and then false is set.
:return: The result status. If overwriting is abandoned, DUPLICATION_ERROR is returned.
"""
pass # native code
def SetMulti(self, overwrite=True, **records):
"""
Sets multiple records of the keyword arguments.
:param overwrite: Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
:param records: Records to store, specified as keyword parameters.
:return: The result status. If there are records avoiding overwriting, DUPLICATION_ERROR is returned.
"""
pass # native code
def SetAndGet(self, key, value, overwrite=True):
"""
Sets a record and get the old value.
:param key: The key of the record.
:param value: The value of the record.
:param overwrite: Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
:return: A pair of the result status and the old value. If the record has not existed when inserting the new record, None is assigned as the value. If not None, the type of the returned old value is the same as the parameter value.
"""
pass # native code
def Remove(self, key):
"""
Removes a record of a key.
:param key: The key of the record.
:return: The result status. If there's no matching record, NOT_FOUND_ERROR is returned.
"""
pass # native code
def RemoveMulti(self, keys):
"""
Removes records of keys.
:param key: The keys of the records.
:return: The result status. If there are missing records, NOT_FOUND_ERROR is returned.
"""
pass # native code
def RemoveAndGet(self, key):
"""
Removes a record and get the value.
:param key: The key of the record.
:return: A pair of the result status and the record value. If the record does not exist, None is assigned as the value. If not None, the type of the returned value is the same as the parameter key.
"""
pass # native code
def Append(self, key, value, delim=""):
"""
Appends data at the end of a record of a key.
:param key: The key of the record.
:param value: The value to append.
:param delim: The delimiter to put after the existing record.
:return: The result status.
If there's no existing record, the value is set without the delimiter.
"""
pass # native code
def AppendMulti(self, delim="", **records):
"""
Appends data to multiple records of the keyword arguments.
:param delim: The delimiter to put after the existing record.
:param records: Records to append, specified as keyword parameters.
:return: The result status.
If there's no existing record, the value is set without the delimiter.
"""
pass # native code
def CompareExchange(self, key, expected, desired):
"""
Compares the value of a record and exchanges if the condition meets.
:param key: The key of the record.
:param expected: The expected value. If it is None, no existing record is expected. If it is ANY_DATA, an existing record with any value is expacted.
:param desired: The desired value. If it is None, the record is to be removed. If it is ANY_DATA, no update is done.
:return: The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.
"""
pass # native code
def CompareExchangeAndGet(self, key, expected, desired):
"""
Does compare-and-exchange and/or gets the old value of the record.
:param key: The key of the record.
:param expected: The expected value. If it is None, no existing record is expected. If it is ANY_DATA, an existing record with any value is expacted.
:param desired: The desired value. If it is None, the record is to be removed. If it is ANY_DATA, no update is done.
:return: A pair of the result status and the.old value of the record. If the condition doesn't meet, the state is INFEASIBLE_ERROR. If there's no existing record, the value is None. If not None, the type of the returned old value is the same as the expected or desired value.
"""
pass # native code
def Increment(self, key, inc=1, init=0, status=None):
"""
Increments the numeric value of a record.
:param key: The key of the record.
:param inc: The incremental value. If it is Utility.INT64MIN, the current value is not changed and a new record is not created.
:param init: The initial value.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The current value, or None on failure.
The record value is stored as an 8-byte big-endian integer. Negative is also supported.
"""
pass # native code
def CompareExchangeMulti(self, expected, desired):
"""
Compares the values of records and exchanges if the condition meets.
:param expected: A sequence of pairs of the record keys and their expected values. If the value is None, no existing record is expected. If the value is ANY_DATA, an existing record with any value is expacted.
:param desired: A sequence of pairs of the record keys and their desired values. If the value is None, the record is to be removed.
:return: The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned.
"""
pass # native code
def Rekey(old_key, new_key, overwrite=True, copying=False):
"""
Changes the key of a record.
:param old_key: The old key of the record.
:param new_key: The new key of the record.
:param overwrite: Whether to overwrite the existing record of the new key.
:param copying: Whether to retain the record of the old key.
:return: The result status. If there's no matching record to the old key, NOT_FOUND_ERROR is returned. If the overwrite flag is false and there is an existing record of the new key, DUPLICATION ERROR is returned.
This method is done atomically. The other threads observe that the record has either the old key or the new key. No intermediate states are observed.
"""
pass # native code
def PopFirst(self, status=None):
"""
Gets the first record and removes it.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: A tuple of the bytes key and the bytes value of the first record. On failure, None is returned.
"""
pass # native code
def PopFirstStr(self, status=None):
"""
Gets the first record as strings and removes it.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: A tuple of the string key and the string value of the first record. On failure, None is returned.
"""
pass # native code
def PushLast(self, value, wtime=None):
"""
Adds a record with a key of the current timestamp.
:param value: The value of the record.
:param wtime: The current wall time used to generate the key. If it is None, the system clock is used.
:return: The result status.
The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds.
"""
pass # native code
def ProcessEach(self, func, writable):
"""
Processes each and every record in the database with an arbitrary function.
:param func: The function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or None if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is None, the record is not modified. If the return value is False (not a false value but the False object), the record is removed.
:param writable: True if the processor can edit the record.
:return: The result status.
The given function is called repeatedly for each record. It is also called once before the iteration and once after the iteration with both the key and the value being None. This method is not available in the concurrent mode because the function cannot be invoked outside the GIL.
"""
pass # native code
def Count(self):
"""
Gets the number of records.
:return: The number of records on success, or None on failure.
"""
pass # native code
def GetFileSize(self):
"""
Gets the current file size of the database.
:return: The current file size of the database, or None on failure.
"""
pass # native code
def GetFilePath(self):
"""
Gets the path of the database file.
:return: The file path of the database, or None on failure.
"""
pass # native code
def GetTimestamp(self):
"""
Gets the timestamp in seconds of the last modified time.
:return: The timestamp of the last modified time, or None on failure.
"""
pass # native code
def Clear(self):
"""
Removes all records.
:return: The result status.
"""
pass # native code
def Rebuild(self, **params):
"""
Rebuilds the entire database.
:param params: Optional keyword parameters.
:return: The result status.
The optional parameters are the same as the Open method. Omitted tuning parameters are kept the same or implicitly optimized.
In addition, HashDBM, TreeDBM, and SkipDBM supports the following parameters.
- skip_broken_records (bool): If true, the operation continues even if there are broken records which can be skipped.
- sync_hard (bool): If true, physical synchronization with the hardware is done before finishing the rebuilt file.
"""
pass # native code
def ShouldBeRebuilt(self):
"""
Checks whether the database should be rebuilt.
:return: True to be optimized or false with no necessity.
"""
pass # native code
def Synchronize(self, hard, **params):
"""
Synchronizes the content of the database to the file system.
:param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
:param params: Optional keyword parameters.
:return: The result status.
Only SkipDBM uses the optional parameters. The "merge" parameter specifies paths of databases to merge, separated by colon. The "reducer" parameter specifies the reducer to apply to records of the same key. "ReduceToFirst", "ReduceToSecond", "ReduceToLast", etc are supported.
"""
pass # native code
def CopyFileData(self, dest_path, sync_hard=False):
"""
Copies the content of the database file to another file.
:param dest_path: A path to the destination file.
:param sync_hard: True to do physical synchronization with the hardware.
:return: The result status.
"""
pass # native code
def Export(self, dest_dbm):
"""
Exports all records to another database.
:param dest_dbm: The destination database.
:return: The result status.
"""
pass # native code
def ExportToFlatRecords(self, dest_file):
"""
Exports all records of a database to a flat record file.
:param dest_file: The file object to write records in.
:return: The result status.
A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration.
"""
pass # native code
def ImportFromFlatRecords(self, src_file):
"""
Imports records to a database from a flat record file.
:param src_file: The file object to read records from.
:return: The result status.
"""
pass # native code
def ExportKeysAsLines(self, dest_file):
"""
Exports the keys of all records as lines to a text file.
:param dest_file: The file object to write keys in.
:return: The result status.
As the exported text file is smaller than the database file, scanning the text file by the search method is often faster than scanning the whole database.
"""
pass # native code
def Inspect(self):
"""
Inspects the database.
:return: A map of property names and their values.
"""
pass # native code
def IsOpen(self):
"""
Checks whether the database is open.
:return: True if the database is open, or false if not.
"""
pass # native code
def IsWritable(self):
"""
Checks whether the database is writable.
:return: True if the database is writable, or false if not.
"""
pass # native code
def IsHealthy(self):
"""
Checks whether the database condition is healthy.
:return: True if the database condition is healthy, or false if not.
"""
pass # native code
def IsOrdered(self):
"""
Checks whether ordered operations are supported.
:return: True if ordered operations are supported, or false if not.
"""
pass # native code
def Search(self, mode, pattern, capacity=0):
"""
Searches the database and get keys which match a pattern.
:param mode: The search mode. "contain" extracts keys containing the pattern. "begin" extracts keys beginning with the pattern. "end" extracts keys ending with the pattern. "regex" extracts keys partially matches the pattern of a regular expression. "edit" extracts keys whose edit distance to the UTF-8 pattern is the least. "editbin" extracts keys whose edit distance to the binary pattern is the least. "containcase", "containword", and "containcaseword" extract keys considering case and word boundary. Ordered databases support "upper" and "lower" which extract keys whose positions are upper/lower than the pattern. "upperinc" and "lowerinc" are their inclusive versions.
:param pattern: The pattern for matching.
:param capacity: The maximum records to obtain. 0 means unlimited.
:return: A list of string keys matching the condition.
"""
pass # native code
def MakeIterator(self):
"""
Makes an iterator for each record.
:return: The iterator for each record.
"""
pass # native code
@classmethod
def RestoreDatabase(cls, old_file_path, new_file_path, class_name="",
end_offset=-1, cipher_key=""):
"""
Restores a broken database as a new healthy database.
:param old_file_path: The path of the broken database.
:param new_file_path: The path of the new database to be created.
:param class_name: The name of the database class. If it is None or empty, the class is guessed from the file extension.
:param end_offset: The exclusive end offset of records to read. Negative means unlimited. 0 means the size when the database is synched or closed properly. Using a positive value is not meaningful if the number of shards is more than one.
:param cipher_key: The encryption key for cipher compressors.
:return: The result status.
"""
class Iterator:
"""
Iterator for each record.
"""
def __init__(self, dbm):
"""
Initializes the iterator.
:param dbm: The database to scan.
"""
pass # native code
def __repr__(self):
"""
Returns A string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns A string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def __next__(self):
"""
Moves the iterator to the next record, to comply to the iterator protocol.
:return: A tuple of The key and the value of the current record.
"""
pass # native code
def First(self):
"""
Initializes the iterator to indicate the first record.
:return: The result status.
Even if there's no record, the operation doesn't fail.
"""
pass # native code
def Last(self):
"""
Initializes the iterator to indicate the last record.
:return: The result status.
Even if there's no record, the operation doesn't fail. This method is suppoerted only by ordered databases.
"""
pass # native code
def Jump(self, key):
"""
Initializes the iterator to indicate a specific record.
:param key: The key of the record to look for.
:return: The result status.
Ordered databases can support "lower bound" jump; If there's no record with the same key, the iterator refers to the first record whose key is greater than the given key. The operation fails with unordered databases if there's no record with the same key.
"""
pass # native code
def JumpLower(self, key, inclusive=False):
"""
Initializes the iterator to indicate the last record whose key is lower than a given key.
:param key: The key to compare with.
:param inclusive: If true, the considtion is inclusive: equal to or lower than the key.
:return: The result status.
Even if there's no matching record, the operation doesn't fail. This method is suppoerted only by ordered databases.
"""
pass # native code
def JumpUpper(self, key, inclusive=False):
"""
Initializes the iterator to indicate the first record whose key is upper than a given key.
:param key: The key to compare with.
:param inclusive: If true, the considtion is inclusive: equal to or upper than the key.
:return: The result status.
Even if there's no matching record, the operation doesn't fail. This method is suppoerted only by ordered databases.
"""
pass # native code
def Next(self):
"""
Moves the iterator to the next record.
:return: The result status.
If the current record is missing, the operation fails. Even if there's no next record, the operation doesn't fail.
"""
pass # native code
def Previous(self):
"""
Moves the iterator to the previous record.
:return: The result status.
If the current record is missing, the operation fails. Even if there's no previous record, the operation doesn't fail. This method is suppoerted only by ordered databases.
"""
pass # native code
def Get(self, status=None):
"""
Gets the key and the value of the current record of the iterator.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: A tuple of the bytes key and the bytes value of the current record. On failure, None is returned.
"""
pass # native code
def GetStr(self, status=None):
"""
Gets the key and the value of the current record of the iterator, as strings.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: A tuple of the string key and the string value of the current record. On failure, None is returned.
"""
pass # native code
def GetKey(self, status=None):
"""
Gets the key of the current record.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The bytes key of the current record or None on failure.
"""
pass # native code
def GetKeyStr(self, status=None):
"""
Gets the key of the current record, as a string.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The string key of the current record or None on failure.
"""
pass # native code
def GetValue(self, status=None):
"""
Gets the value of the current record.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The bytes value of the current record or None on failure.
"""
pass # native code
def GetValueStr(self, status=None):
"""
Gets the value of the current record, as a string.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The string value of the current record or None on failure.
"""
pass # native code
def Set(self, value):
"""
Sets the value of the current record.
:param value: The value of the record.
:return: The result status.
"""
pass # native code
def Remove(self):
"""
Removes the current record.
:return: The result status.
"""
pass # native code
def Step(self, status=None):
"""
Gets the current record and moves the iterator to the next record.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: A tuple of the bytes key and the bytes value of the current record. On failure, None is returned.
"""
pass # native code
def StepStr(self, status=None):
"""
Gets the current record and moves the iterator to the next record, as strings.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: A tuple of the string key and the string value of the current record. On failure, None is returned.
"""
pass # native code
class AsyncDBM:
"""
Asynchronous database manager adapter.
This class is a wrapper of DBM for asynchronous operations. A task queue with a thread pool is used inside. Every method except for the constructor and the destructor is run by a thread in the thread pool and the result is set in the future oject of the return value. The caller can ignore the future object if it is not necessary. The Destruct method waits for all tasks to be done. Therefore, the destructor should be called before the database is closed.
"""
def __init__(self, dbm, num_worker_threads):
"""
Sets up the task queue.
:param dbm: A database object which has been opened.
:param num_worker_threads: The number of threads in the internal thread pool.
"""
pass # native code
def __repr__(self):
"""
Returns A string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns a string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def Destruct():
"""
Destructs the asynchronous database adapter.
This method waits for all tasks to be done.
"""
def Get(self, key):
"""
Gets the value of a record of a key.
:param key: The key of the record.
:return: The future for the result status and the bytes value of the matching record.
"""
pass # native code
def GetStr(self, key):
"""
Gets the value of a record of a key, as a string.
:param key: The key of the record.
:return: The future for the result status and the string value of the matching record.
"""
pass # native code
def GetMulti(self, *keys):
"""
Gets the values of multiple records of keys.
:param keys: The keys of records to retrieve.
:return: The future for the result status and a map of retrieved records. Keys which don't match existing records are ignored.
"""
pass # native code
def GetMultiStr(self, *keys):
"""
Gets the values of multiple records of keys, as strings.
:param keys: The keys of records to retrieve.
:return: The future for the result status and a map of retrieved records. Keys which don't match existing records are ignored.
"""
pass # native code
def Set(self, key, value, overwrite=True):
"""
Sets a record of a key and a value.
:param key: The key of the record.
:param value: The value of the record.
:param overwrite: Whether to overwrite the existing value. It can be omitted and then false is set.
:return: The future for the result status. If overwriting is abandoned, DUPLICATION_ERROR is set.
"""
pass # native code
def SetMulti(self, overwrite=True, **records):
"""
Sets multiple records of the keyword arguments.
:param overwrite: Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned.
:param records: Records to store, specified as keyword parameters.
:return: The future for the result status. If overwriting is abandoned, DUPLICATION_ERROR is set.
"""
pass # native code
def Append(self, key, value, delim=""):
"""
Appends data at the end of a record of a key.
:param key: The key of the record.
:param value: The value to append.
:param delim: The delimiter to put after the existing record.
:return: The future for the result status.
If there's no existing record, the value is set without the delimiter.
"""
pass # native code
def AppendMulti(self, delim="", **records):
"""
Appends data to multiple records of the keyword arguments.
:param delim: The delimiter to put after the existing record.
:param records: Records to append, specified as keyword parameters.
:return: The future for the result status.
If there's no existing record, the value is set without the delimiter.
"""
pass # native code
def CompareExchange(self, key, expected, desired):
"""
Compares the value of a record and exchanges if the condition meets.
:param key: The key of the record.
:param expected: The expected value. If it is None, no existing record is expected. If it is DBM.ANY_DATA, an existing record with any value is expacted.
:param desired: The desired value. If it is None, the record is to be removed. If it is None, the record is to be removed. If it is DBM.ANY_DATA, no update is done.
:return: The future for the result status. If the condition doesn't meet, INFEASIBLE_ERROR is set.
"""
pass # native code
def Increment(self, key, inc=1, init=0):
"""
Increments the numeric value of a record.
:param key: The key of the record.
:param inc: The incremental value. If it is Utility.INT64MIN, the current value is not changed and a new record is not created.
:param init: The initial value.
:return: The future for the result status and the current value.
The record value is stored as an 8-byte big-endian integer. Negative is also supported.
"""
pass # native code
def ProcessMulti(self, key_func_pairs, writable):
"""
Processes multiple records with arbitrary functions.
:param key_func_pairs: A list of pairs of keys and their functions. The first parameter of the function is the key of the record. The second parameter is the value of the existing record, or None if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is None, the record is not modified. If the return value is False (not a false value but the False object), the record is removed.
:param writable: True if the processors can edit the record.
:return: The result status.
This method is not available in the concurrent mode because the function cannot be invoked outside the GIL.
"""
pass # native code
def CompareExchangeMulti(self, expected, desired):
"""
Compares the values of records and exchanges if the condition meets.
:param expected: A sequence of pairs of the record keys and their expected values. If the value is None, no existing record is expected. If the value is DBM.ANY_DATA, an existing record with any value is expacted.
:param desired: A sequence of pairs of the record keys and their desired values. If the value is None, the record is to be removed.
:return: The future for the result status. If the condition doesn't meet, INFEASIBLE_ERROR is set.
"""
pass # native code
def Rekey(old_key, new_key, overwrite=True, copying=False):
"""
Changes the key of a record.
:param old_key: The old key of the record.
:param new_key: The new key of the record.
:param overwrite: Whether to overwrite the existing record of the new key.
:param copying: Whether to retain the record of the old key.
:return: The future for the result status. If there's no matching record to the old key, NOT_FOUND_ERROR is set. If the overwrite flag is false and there is an existing record of the new key, DUPLICATION ERROR is set.
This method is done atomically. The other threads observe that the record has either the old key or the new key. No intermediate states are observed.
"""
pass # native code
def PopFirst(self):
"""
Gets the first record and removes it.
:return: The future for a tuple of the result status, the bytes key, and the bytes value of the first record.
"""
pass # native code
def PopFirstStr(self):
"""
Gets the first record as strings and removes it.
:return: The future for a tuple of the result status, the string key, and the string value of the first record.
"""
pass # native code
def PushLast(self, value, wtime=None):
"""
Adds a record with a key of the current timestamp.
:param value: The value of the record.
:param wtime: The current wall time used to generate the key. If it is None, the system clock is used.
:return: The future for the result status.
The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds.
"""
pass # native code
def Clear(self):
"""
Removes all records.
:return: The future for the result status.
"""
pass # native code
def Rebuild(self, **params):
"""
Rebuilds the entire database.
:param params: Optional keyword parameters.
:return: The future for the result status.
The parameters work in the same way as with DBM::Rebuild.
"""
pass # native code
def Synchronize(self, hard, **params):
"""
Synchronizes the content of the database to the file system.
:param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
:param params: Optional keyword parameters.
:return: The future for the result status.
The parameters work in the same way as with DBM::Synchronize.
"""
pass # native code
def CopyFileData(self, dest_path, sync_hard=False):
"""
Copies the content of the database file to another file.
:param dest_path: A path to the destination file.
:param sync_hard: True to do physical synchronization with the hardware.
:return: The future for the result status.
"""
pass # native code
def Export(self, dest_dbm):
"""
Exports all records to another database.
:param dest_dbm: The destination database. The lefetime of the database object must last until the task finishes.
:return: The future for the result status.
"""
pass # native code
def ExportToFlatRecords(self, dest_file):
"""
Exports all records of a database to a flat record file.
:param dest_file: The file object to write records in. The lefetime of the file object must last until the task finishes.
:return: The future for the result status.
A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration.
"""
pass # native code
def ImportFromFlatRecords(self, src_file):
"""
Imports records to a database from a flat record file.
:param src_file: The file object to read records from. The lefetime of the file object must last until the task finishes.
:return: The future for the result status.
"""
pass # native code
def Search(self, mode, pattern, capacity=0):
"""
Searches the database and get keys which match a pattern.
:param mode: The search mode. "contain" extracts keys containing the pattern. "begin" extracts keys beginning with the pattern. "end" extracts keys ending with the pattern. "regex" extracts keys partially matches the pattern of a regular expression. "edit" extracts keys whose edit distance to the UTF-8 pattern is the least. "editbin" extracts keys whose edit distance to the binary pattern is the least.
:param pattern: The pattern for matching.
:param capacity: The maximum records to obtain. 0 means unlimited.
:return: The future for the result status and a list of keys matching the condition.
"""
pass # native code
class File:
"""
Generic file implementation.
All operations except for "open" and "close" are thread-safe; Multiple threads can access the same file concurrently. You can specify a concrete class when you call the "open" method. Every opened file must be closed explicitly by the "close" method to avoid data corruption.
"""
def __init__(self):
"""
Initializes the file object.
"""
pass # native code
def __repr__(self):
"""
Returns A string representation of the object.
:return: The string representation of the object.
"""
pass # native code
def __str__(self):
"""
Returns A string representation of the content.
:return: The string representation of the content.
"""
pass # native code
def Open(self, path, writable, **params):
"""
Opens a file.
:param path: A path of the file.
:param writable: If true, the file is writable. If false, it is read-only.
:param params: Optional keyword parameters.
:return: The result status.
The optional parameters can include an option for the concurrency tuning. By default, database operatins are done under the GIL (Global Interpreter Lock), which means that database operations are not done concurrently even if you use multiple threads. If the "concurrent" parameter is true, database operations are done outside the GIL, which means that database operations can be done concurrently if you use multiple threads. However, the downside is that swapping thread data is costly so the actual throughput is often worse in the concurrent mode than in the normal mode. Therefore, the concurrent mode should be used only if the database is huge and it can cause blocking of threads in multi-thread usage.
The optional parameters can include options for the file opening operation.
- truncate (bool): True to truncate the file.
- no_create (bool): True to omit file creation.
- no_wait (bool): True to fail if the file is locked by another process.
- no_lock (bool): True to omit file locking.
- sync_hard (bool): True to do physical synchronization when closing.
The optional parameter "file" specifies the internal file implementation class. The default file class is "MemoryMapAtomicFile". The other supported classes are "StdFile", "MemoryMapAtomicFile", "PositionalParallelFile", and "PositionalAtomicFile".
For the file "PositionalParallelFile" and "PositionalAtomicFile", these optional parameters are supported.
- block_size (int): The block size to which all blocks should be aligned.
- access_options (str): Values separated by colon. "direct" for direct I/O. "sync" for synchrnizing I/O, "padding" for file size alignment by padding, "pagecache" for the mini page cache in the process.
"""
pass # native code
def Close(self):
"""
Closes the file.
:return: The result status.
"""
pass # native code
def Read(self, off, size, status=None):
"""
Reads data.
:param off: The offset of a source region.
:param size: The size to be read.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The bytes value of the read data or None on failure.
"""
pass # native code
def ReadStr(self, off, size, status=None):
"""
Reads data as a string.
:param off: The offset of a source region.
:param size: The size to be read.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The string value of the read data or None on failure.
"""
pass # native code
def Write(self, off, data):
"""
Writes data.
:param off: The offset of the destination region.
:param data: The data to write.
:return: The result status.
"""
pass # native code
def Append(self, data, status=None):
"""
Appends data at the end of the file.
:param data: The data to write.
:param status: A status object to which the result status is assigned. It can be omitted.
:return: The offset at which the data has been put, or None on failure.
"""
pass # native code
def Truncate(self, size):
"""
Truncates the file.
:param size: The new size of the file.
:return: The result status.
If the file is shrunk, data after the new file end is discarded. If the file is expanded, null codes are filled after the old file end.
"""
pass # native code
def Synchronize(self, hard, off=0, size=0):
"""
Synchronizes the content of the file to the file system.
:param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
:param off: The offset of the region to be synchronized.
:param size: The size of the region to be synchronized. If it is zero, the length to the end of file is specified.
:return: The result status.
The pysical file size can be larger than the logical size in order to improve performance by reducing frequency of allocation. Thus, you should call this function before accessing the file with external tools.
"""
pass # native code
def GetSize(self):
"""
Gets the size of the file.
:return: The size of the file or None on failure.
"""
pass # native code
def GetPath(self):
"""
Gets the path of the file.
:return: The path of the file or None on failure.
"""
pass # native code
def Search(self, mode, pattern, capacity=0):
"""
Searches the file and get lines which match a pattern.
:param mode: The search mode. "contain" extracts lines containing the pattern. "begin" extracts lines beginning with the pattern. "end" extracts lines ending with the pattern. "regex" extracts lines partially matches the pattern of a regular expression. "edit" extracts lines whose edit distance to the UTF-8 pattern is the least. "editbin" extracts lines whose edit distance to the binary pattern is the least.
:param pattern: The pattern for matching.
:param capacity: The maximum records to obtain. 0 means unlimited.
:return: A list of lines matching the condition.
"""
pass # native code
# END OF FILE
tkrzw-python-0.1.30/tkrzw.cc 0000664 0001750 0001750 00000446774 14402705440 015015 0 ustar mikio mikio /*************************************************************************************************
* Python binding of Tkrzw
*
* Copyright 2020 Google LLC
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file
* except in compliance with the License. You may obtain a copy of the License at
* https://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software distributed under the
* License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
* either express or implied. See the License for the specific language governing permissions
* and limitations under the License.
*************************************************************************************************/
#include
#include
#include