tkrzw-python-0.1.32/0000775000175000017500000000000014626642264013315 5ustar mikiomikiotkrzw-python-0.1.32/README0000664000175000017500000000107014626642264014173 0ustar mikiomikio=============================================================== Python interface of Tkrzw ================================================================ Please read the following documents. COPYING - license CONTRIBUTING.md - how to contribute You should install the latest version of Tkrzw to make sure the compatibility. To build the library, Python 3.6 or later version is required. Then, run these commands. make make check sudo make install See https://dbmx.net/tkrzw/api-python/ for details of the API. Thanks. == END OF FILE == tkrzw-python-0.1.32/example6.py0000775000175000017500000000353214626642264015416 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Example for secondary index # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import tkrzw # Opens the index. index = tkrzw.Index() index.Open("casket.tkt", True, truncate=True, num_buckets=100).OrDie() # Adds records to the index. # The key is a division name and the value is person name. index.Add("general", "anne").OrDie() index.Add("general", "matthew").OrDie() index.Add("general", "marilla").OrDie() index.Add("sales", "gilbert").OrDie() # Anne moves to the sales division. index.Remove("general", "anne").OrDie() index.Add("sales", "anne").OrDie() # Prints all members for each division. for division in ["general", "sales"]: print(division) members = index.GetValuesStr(division) for member in members: print(" -- " + member) # Prints every record by iterator. iter = index.MakeIterator() iter.First() while True: record = iter.GetStr() if not record: break print(record[0] + ": " + record[1]) iter.Next() # Prints every record by the iterable protocol. for key, value in index: print(key.decode() + ": " + value.decode()) # Closes the index index.Close().OrDie() # END OF FILE tkrzw-python-0.1.32/index.rst0000664000175000017500000003664414626642264015173 0ustar mikiomikioModule and Classes ================== .. autosummary:: :nosignatures: tkrzw tkrzw.Utility tkrzw.Status tkrzw.StatusException tkrzw.DBM tkrzw.Iterator tkrzw.Future tkrzw.AsyncDBM tkrzw.File tkrzw.Index tkrzw.IndexIterator Introduction ============ DBM (Database Manager) is a concept to store an associative array on a permanent storage. In other words, DBM allows an application program to store key-value pairs in a file and reuse them later. Each of keys and values is a string or a sequence of bytes. A key must be unique within the database and a value is associated to it. You can retrieve a stored record with its key very quickly. Thanks to simple structure of DBM, its performance can be extremely high. Tkrzw is a library implementing DBM with various algorithms. It features high degrees of performance, concurrency, scalability and durability. The following data structures are provided. - HashDBM : File datatabase manager implementation based on hash table. - TreeDBM : File datatabase manager implementation based on B+ tree. - SkipDBM : File datatabase manager implementation based on skip list. - TinyDBM : On-memory datatabase manager implementation based on hash table. - BabyDBM : On-memory datatabase manager implementation based on B+ tree. - CacheDBM : On-memory datatabase manager implementation with LRU deletion. - StdHashDBM : On-memory DBM implementations using std::unordered_map. - StdTreeDBM : On-memory DBM implementations using std::map. Whereas Tkrzw is C++ library, this package provides its Python interface. All above data structures are available via one adapter class ":class:`DBM`". Read the `homepage `_ for details. DBM stores key-value pairs of strings. Each string is represented as bytes in Python. You can specify any type of objects as keys and values if they can be converted into strings, which are "encoded" into bytes. When you retreive the value of a record, the type is determined according to the method: Get for bytes, GetStr for string, or [] for the same type as the key. Symbols of the module "tkrzw" should be imported in each source file of application programs.:: import tkrzw An instance of the class ":class:`DBM`" is used in order to handle a database. You can store, delete, and retrieve records with the instance. The result status of each operation is represented by an object of the class ":class:`Status`". Iterator to access access each record is implemented by the class ":class:`Iterator`". Installation ============ Install the latest version of Tkrzw beforehand and get the package of the Python binding of Tkrzw. Python 3.6 or later is required to use this package. Enter the directory of the extracted package then perform installation. If your system uses another command than the "python3" command, edit the Makefile beforehand.:: make make check sudo make install Example ======= The following code is a typical example to use a database. A DBM object can be used like a dictionary object. As DBM implements the generic iterator protocol, you can access each record with the "for" loop.:: import tkrzw # Prepares the database. dbm = tkrzw.DBM() dbm.Open("casket.tkh", True, truncate=True, num_buckets=100) # Sets records. # If the operation fails, a runtime exception is raised. # Keys and values are implicitly converted into bytes. dbm["first"] = "hop" dbm["second"] = "step" dbm["third"] = "jump" # Retrieves record values. # If the operation fails, a runtime exception is raised. # Retrieved values are strings if keys are strings. print(dbm["first"]) print(dbm["second"]) print(dbm["third"]) try: print(dbm["fourth"]) except tkrzw.StatusException as e: print(repr(e)) # Checks and deletes a record. if "first" in dbm: del dbm["first"] # Traverses records. # Retrieved keys and values are always bytes so we decode them. for key, value in dbm: print(key.decode(), value.decode()) # Closes the database. dbm.Close() The following code is a more complex example. Resources of DBM and Iterator are bound to their objects so when the refenrece count becomes zero, resources are released. Even if the database is not closed, the destructor closes it implicitly. The method "OrDie" throws an exception on failure so it is useful for checking errors.:: import tkrzw # Prepares the database. # Options are given by dictionary expansion. # All methods except for [] and []= don't raise exceptions. dbm = tkrzw.DBM() open_params = { "max_page_size": 4080, "max_branches": 256, "key_comparator": "decimal", "concurrent": True, "truncate": True, } status = dbm.Open("casket.tkt", True, **open_params) if not status.IsOK(): raise tkrzw.StatusException(status) # Sets records. # The method OrDie raises a runtime error on failure. dbm.Set(1, "hop").OrDie() dbm.Set(2, "step").OrDie() dbm.Set(3, "jump").OrDie() # Retrieves records without checking errors. # On failure, the return value is None. print(dbm.GetStr(1)) print(dbm.GetStr(2)) print(dbm.GetStr(3)) print(dbm.GetStr(4)) # To know the status of retrieval, give a status object to Get. # You can compare a status object and a status code directly. status = tkrzw.Status() value = dbm.GetStr(1, status) print("status: " + str(status)) if status == tkrzw.Status.SUCCESS: print("value: " + value) # Rebuilds the database. # Almost the same options as the Open method can be given. dbm.Rebuild(align_pow=0, max_page_size=1024).OrDie() # Traverses records with an iterator. iter = dbm.MakeIterator() iter.First() while True: status = tkrzw.Status() record = iter.GetStr(status) if not status.IsOK(): break print(record[0], record[1]) iter.Next() # Closes the database. dbm.Close().OrDie() The following code is examples to use a tree database which handles numeric keys with various key comparators. For integers, there are two ways: 1) to use the default comparator and serialize integer keys into byte sequences, or 2) to use the decimal integer comparator and represent keys as decimal integers like "123". For real numbers, there are also two ways: 1) to use the decimal real number comparator and represents keys as decimal real numbers like "123.45", or 2) to use the big-endian floating-point numbers comparator and serialize floating-point numbers into byte sequences.:: import tkrzw # Opens a new database with the default key comparator (LexicalKeyComparator). dbm = tkrzw.DBM() open_params = { "truncate": True, } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a big-endian binary of an integer. # e.g: "\x00\x00\x00\x00\x00\x00\x00\x31" -> "hop" dbm.Set(tkrzw.Utility.SerializeInt(1), "hop").OrDie() dbm.Set(tkrzw.Utility.SerializeInt(256), "step").OrDie() dbm.Set(tkrzw.Utility.SerializeInt(32), "jump").OrDie() # Gets records with the key being a decimal string of an integer. print(dbm.GetStr(tkrzw.Utility.SerializeInt(1))) print(dbm.GetStr(tkrzw.Utility.SerializeInt(256))) print(dbm.GetStr(tkrzw.Utility.SerializeInt(32))) # Lists up all records, restoring keys into integers. iter = dbm.MakeIterator() iter.First() while True: record = iter.Get() if not record: break print(str(tkrzw.Utility.DeserializeInt(record[0])) + ": " + record[1].decode()) iter.Next() # Closes the database. dbm.Close().OrDie() # Opens a new database with the decimal integer comparator. open_params = { "truncate": True, "key_comparator": "Decimal", } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a decimal string of an integer. # e.g: "1" -> "hop" dbm.Set("1", "hop").OrDie() dbm.Set("256", "step").OrDie() dbm.Set("32", "jump").OrDie() # Gets records with the key being a decimal string of an integer. print(dbm.GetStr("1")) print(dbm.GetStr("256")) print(dbm.GetStr("32")) # Lists up all records, restoring keys into integers. iter = dbm.MakeIterator() iter.First() while True: record = iter.GetStr() if not record: break print("{:d}: {}".format(int(record[0]), record[1])) iter.Next() # Closes the database. dbm.Close().OrDie() # Opens a new database with the decimal real number comparator. open_params = { "truncate": True, "key_comparator": "RealNumber", } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a decimal string of a real number. # e.g: "1.5" -> "hop" dbm.Set("1.5", "hop").OrDie() dbm.Set("256.5", "step").OrDie() dbm.Set("32.5", "jump").OrDie() # Gets records with the key being a decimal string of a real number. print(dbm.GetStr("1.5")) print(dbm.GetStr("256.5")) print(dbm.GetStr("32.5")) # Lists up all records, restoring keys into floating-point numbers. iter = dbm.MakeIterator() iter.First() while True: record = iter.GetStr() if not record: break print("{:.3f}: {}".format(float(record[0]), record[1])) iter.Next() # Closes the database. dbm.Close().OrDie() # Opens a new database with the big-endian floating-point numbers comparator. open_params = { "truncate": True, "key_comparator": "FloatBigEndian", } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a big-endian binary of a floating-point number. # e.g: "\x3F\xF8\x00\x00\x00\x00\x00\x00" -> "hop" dbm.Set(tkrzw.Utility.SerializeFloat(1.5), "hop").OrDie() dbm.Set(tkrzw.Utility.SerializeFloat(256.5), "step").OrDie() dbm.Set(tkrzw.Utility.SerializeFloat(32.5), "jump").OrDie() # Gets records with the key being a big-endian binary of a floating-point number. print(dbm.GetStr(tkrzw.Utility.SerializeFloat(1.5))) print(dbm.GetStr(tkrzw.Utility.SerializeFloat(256.5))) print(dbm.GetStr(tkrzw.Utility.SerializeFloat(32.5))) # Lists up all records, restoring keys into floating-point numbers. iter = dbm.MakeIterator() iter.First() while True: record = iter.Get() if not record: break print("{:.3f}: {}".format(tkrzw.Utility.DeserializeFloat(record[0]), record[1].decode())) iter.Next() # Closes the database. dbm.Close().OrDie() The following code is a typical example of coroutine usage. The AsyncDBM class manages a thread pool and handles database operations in the background in parallel. Each Method of AsyncDBM returns a Future object to monitor the result. The Future class implements the awaitable protocol so that the instance is usable with the "await" sentence to await the operation while yielding the execution ownership.:: import asyncio import tkrzw async def main(): # Prepares the database. dbm = tkrzw.DBM() dbm.Open("casket.tkh", True, truncate=True, num_buckets=100) # Prepares the asynchronous adapter with 4 worker threads. adbm = tkrzw.AsyncDBM(dbm, 4) # Executes the Set method asynchronously. future = adbm.Set("hello", "world") # Does something in the foreground. print("Setting a record") # Checks the result after awaiting the Set operation. # Calling Future#Get doesn't yield the coroutine ownership. status = future.Get() if status != tkrzw.Status.SUCCESS: print("ERROR: " + str(status)) # Executes the Get method asynchronously. future = adbm.GetStr("hello") # Does something in the foreground. print("Getting a record") # Awaits the operation while yielding the execution ownership. await future status, value = future.Get() if status == tkrzw.Status.SUCCESS: print("VALUE: " + value) # Releases the asynchronous adapter. adbm.Destruct() # Closes the database. dbm.Close() asyncio.run(main()) The following code uses Process, ProcessMulti, and ProcessEach functions which take callback functions to process the record efficiently. Process is useful to update a record atomically according to the current value. ProcessEach is useful to access every record in the most efficient way.:: import tkrzw import re # Opens the database. dbm = tkrzw.DBM() dbm.Open("casket.tkh", True, truncate=True, num_buckets=1000) # Sets records with lambda functions. dbm.Process("doc-1", lambda key, value: "Tokyo is the capital city of Japan.", True) dbm.Process("doc-2", lambda key, value: "Is she living in Tokyo, Japan?", True) dbm.Process("doc-3", lambda key, value: "She must leave Tokyo!", True) # Lowers record values. def Lower(key, value): # If no matching record, None is given as the value. if not value: return None # Sets the new value. # Note that the key and the value are a "bytes" object. return value.decode().lower() dbm.Process("doc-1", Lower, True) dbm.Process("doc-2", Lower, True) dbm.Process("non-existent", Lower, True) # Does the same thing with a lambda function. dbm.Process("doc-3", lambda key, value: value.decode().lower() if value else None, True) # If you don't update the record, set the third parameter to false. dbm.Process("doc-3", lambda key, value: print(key, value), False) # Adds multiple records at once. dbm.ProcessMulti([ ("doc-4", lambda key, value: "Tokyo Go!"), ("doc-5", lambda key, value: "Japan Go!")], True) # Modifies multiple records at once. dbm.ProcessMulti([("doc-4", Lower), ("doc-5", Lower)], True) # Checks the whole content. # This uses an external iterator and is relavively slow. for key, value in dbm: print(key.decode(), value.decode()) # Function for word counting. word_counts = {} def WordCounter(key, value): if not key: return value = value.decode() words = [x for x in re.split(r"\W+", value) if x] for word in words: word_counts[word] = (word_counts.get(word) or 0) + 1 # The second parameter should be false if the value is not updated. dbm.ProcessEach(WordCounter, False) print(word_counts) # Returning False by the callbacks removes the record. dbm.Process("doc-1", lambda key, value: False, True) print(dbm.Count()) dbm.ProcessMulti([("doc-2", lambda key, value: False), ("doc-3", lambda key, value: False)], True) print(dbm.Count()) dbm.ProcessEach(lambda key, value: False, True) print(dbm.Count()) # Closes the database. dbm.Close() The following code is an example to use a secondary index, which is useful to organize records by non-primary keys.:: import tkrzw # Opens the database. index = tkrzw.Index() index.Open("casket.tkt", True, truncate=True, num_buckets=100).OrDie() # Adds records to the index. # The key is a division name and the value is person name. index.Add("general", "anne").OrDie() index.Add("general", "matthew").OrDie() index.Add("general", "marilla").OrDie() index.Add("sales", "gilbert").OrDie() # Anne moves to the sales division. index.Remove("general", "anne").OrDie() index.Add("sales", "anne").OrDie() # Prints all members for each division. for division in ["general", "sales"]: print(division) members = index.GetValuesStr(division) for member in members: print(" -- " + member) # Prints every record by iterator. iter = index.MakeIterator() iter.First() while True: record = iter.GetStr() if not record: break print(record[0] + ": " + record[1]) iter.Next() # Prints every record by the iterable protocol. for key, value in index: print(key.decode() + ": " + value.decode()) # Closes the index index.Close().OrDie() Indices and tables ================== .. toctree:: :maxdepth: 4 :caption: Contents: tkrzw * :ref:`genindex` * :ref:`modindex` * :ref:`search` tkrzw-python-0.1.32/example4.py0000775000175000017500000000366714626642264015425 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Example for the asynchronous API # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import asyncio import tkrzw async def main(): # Prepares the database. dbm = tkrzw.DBM() dbm.Open("casket.tkh", True, truncate=True, num_buckets=100) # Prepares the asynchronous adapter with 4 worker threads. adbm = tkrzw.AsyncDBM(dbm, 4) # Executes the Set method asynchronously. future = adbm.Set("hello", "world") # Does something in the foreground. print("Setting a record") # Checks the result after awaiting the Set operation. # Calling Future#Get doesn't yield the coroutine ownership. status = future.Get() if status != tkrzw.Status.SUCCESS: print("ERROR: " + str(status)) # Executes the Get method asynchronously. future = adbm.GetStr("hello") # Does something in the foreground. print("Getting a record") # Awaits the operation while yielding the coroutine ownership. await future status, value = future.Get() if status == tkrzw.Status.SUCCESS: print("VALUE: " + value) # Releases the asynchronous adapter. adbm.Destruct() # Closes the database. dbm.Close() asyncio.run(main()) # END OF FILE tkrzw-python-0.1.32/example2.py0000775000175000017500000000442114626642264015410 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Example for basic usage of the hash database # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import tkrzw # Prepares the database. # Options are given by dictionary expansion. # All methods except for [] and []= don't raise exceptions. dbm = tkrzw.DBM() open_params = { "max_page_size": 4080, "max_branches": 256, "key_comparator": "decimal", "concurrent": True, "truncate": True, } status = dbm.Open("casket.tkt", True, **open_params) if not status.IsOK(): raise tkrzw.StatusException(status) # Sets records. # The method OrDie raises a runtime error on failure. dbm.Set(1, "hop").OrDie() dbm.Set(2, "step").OrDie() dbm.Set(3, "jump").OrDie() # Retrieves records without checking errors. # On failure, the return value is None. print(dbm.GetStr(1)) print(dbm.GetStr(2)) print(dbm.GetStr(3)) print(dbm.GetStr(4)) # To know the status of retrieval, give a status object to Get. # You can compare a status object and a status code directly. status = tkrzw.Status() value = dbm.GetStr(1, status) print("status: " + str(status)) if status == tkrzw.Status.SUCCESS: print("value: " + value) # Rebuilds the database. # Almost the same options as the Open method can be given. dbm.Rebuild(align_pow=0, max_page_size=1024).OrDie() # Traverses records with an iterator. iter = dbm.MakeIterator() iter.First() while True: status = tkrzw.Status() record = iter.GetStr(status) if not status.IsOK(): break print(record[0], record[1]) iter.Next() # Closes the database. dbm.Close().OrDie() # END OF FILE tkrzw-python-0.1.32/setup.py0000664000175000017500000000776514626642264015046 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- # -------------------------------------------------------------------------------------------------- # Building configurations # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. # -------------------------------------------------------------------------------------------------- import os import platform import subprocess import sys from setuptools import Extension, setup package_name = "tkrzw" package_version = "0.1.30" package_description = "a set of implementations of DBM" package_author = "Mikio Hirabayashi" package_author_email = "hirarin@gmail.com" package_url = "http://dbmx.net/tkrzw/" module_name = "tkrzw" keywords = "tkrzw dbm database performance concurrency scalability durability" classifiers = [ "Development Status :: 4 - Beta", "Topic :: Database", "Topic :: Database :: Database Engines/Servers", "Topic :: Software Development :: Libraries", "Programming Language :: Python :: 3", "Programming Language :: Python :: Implementation :: CPython", ] long_description = """ Python interface of Tkrzw ========================= [Tkrzw](https://dbmx.net/tkrzw/) is a C++ library implementing DBM with various algorithms. It features high degrees of performance, concurrency, scalability and durability. We build the following wheels: - musllinux_1_2_x86_64 - manylinux_2_27_x86_64 - TODO: macos - TODO: windows Issues ------ - Please report issues related to the build and the wheels to [tkrzw-wheels](https://github.com/ganwell/tkrzw-wheels) - Please report issues related to the binding to [tkrzw-python](https://github.com/estraier/tkrzw-python) Please read the following documents. - [COPYING](https://github.com/estraier/tkrzw-python/blob/master/COPYING) - [CONTRIBUTING.md](https://github.com/estraier/tkrzw-python/blob/master/CONTRIBUTING.md) """.strip() extra_compile_args = ["-std=c++17", "-Wall"] sources = ["tkrzw.cc"] ld_paths = ["/usr/local/lib"] os.environ["LD_LIBRARY_PATH"] = os.environ.get("LD_LIBRARY_PATH", "") + ":".join( ld_paths ) def parse_build_flags(cmd_args): result = {} try: output = subprocess.check_output(cmd_args).strip().decode() for item in output.split(): flag = item[0:2] result.setdefault(flag, []).append(item[2:]) except FileNotFoundError: sys.stderr.write(f"Warning: {cmd_args[0]} not found") return result # Parse include dirs flags = parse_build_flags(["tkrzw_build_util", "config", "-i"]) include_dirs = flags.get("-I", []) or ["/usr/local/include"] # Parse library dirs and libraries flags = parse_build_flags(["tkrzw_build_util", "config", "-l"]) library_dirs = flags.get("-L", ["/usr/local/lib"]) libraries = flags.get("-l", []) if not libraries: if platform.system() == "Darwin": libraries = ["tkrzw", "stdc++", "pthread", "m", "c"] else: libraries = ["tkrzw", "stdc++", "rt", "atomic", "pthread", "m", "c"] setup( name=package_name, version=package_version, description=package_description, author=package_author, author_email=package_author_email, url=package_url, ext_modules=[ Extension( module_name, include_dirs=include_dirs, extra_compile_args=extra_compile_args, sources=sources, library_dirs=library_dirs, libraries=libraries, ), ], keywords=keywords, classifiers=classifiers, long_description=long_description, long_description_content_type="text/markdown", ) # END OF FILE tkrzw-python-0.1.32/example5.py0000775000175000017500000000601114626642264015410 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Example for process methods # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import tkrzw import re # Opens the database. dbm = tkrzw.DBM() dbm.Open("casket.tkh", True, truncate=True, num_buckets=1000) # Sets records with lambda functions. dbm.Process("doc-1", lambda key, value: "Tokyo is the capital city of Japan.", True) dbm.Process("doc-2", lambda key, value: "Is she living in Tokyo, Japan?", True) dbm.Process("doc-3", lambda key, value: "She must leave Tokyo!", True) # Lowers record values. def Lower(key, value): # If no matching record, None is given as the value. if not value: return None # Sets the new value. # Note that the key and the value are a "bytes" object. return value.decode().lower() dbm.Process("doc-1", Lower, True) dbm.Process("doc-2", Lower, True) dbm.Process("non-existent", Lower, True) # Does the same thing with a lambda function. dbm.Process("doc-3", lambda key, value: value.decode().lower() if value else None, True) # If you don't update the record, set the third parameter to false. dbm.Process("doc-3", lambda key, value: print(key, value), False) # Adds multiple records at once. dbm.ProcessMulti([ ("doc-4", lambda key, value: "Tokyo Go!"), ("doc-5", lambda key, value: "Japan Go!")], True) # Modifies multiple records at once. dbm.ProcessMulti([("doc-4", Lower), ("doc-5", Lower)], True) # Checks the whole content. # This uses an external iterator and is relavively slow. for key, value in dbm: print(key.decode(), value.decode()) # Function for word counting. word_counts = {} def WordCounter(key, value): if not key: return value = value.decode() words = [x for x in re.split(r"\W+", value) if x] for word in words: word_counts[word] = (word_counts.get(word) or 0) + 1 # The second parameter should be false if the value is not updated. dbm.ProcessEach(WordCounter, False) print(word_counts) # Returning False by the callbacks removes the record. dbm.Process("doc-1", lambda key, value: False, True) print(dbm.Count()) dbm.ProcessMulti([("doc-2", lambda key, value: False), ("doc-3", lambda key, value: False)], True) print(dbm.Count()) dbm.ProcessEach(lambda key, value: False, True) print(dbm.Count()) # Closes the database. dbm.Close() # END OF FILE tkrzw-python-0.1.32/test.py0000775000175000017500000012647314626642264014666 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Test cases # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import asyncio import math import os import random import re import shutil import sys import tempfile import threading import time import unittest from tkrzw import * # Unit testing framework. class TestTkrzw(unittest.TestCase): # Prepares resources. def setUp(self): tmp_prefix = "tkrzw-python-" self.test_dir = tempfile.mkdtemp(prefix=tmp_prefix) # Cleanups resources. def tearDown(self): shutil.rmtree(self.test_dir) # Makes a temporary path. def _make_tmp_path(self, name): return os.path.join(self.test_dir, name) # Utility tests. def testUtility(self): self.assertTrue(re.search(r"^\d+.\d+.\d+$", Utility.VERSION)) self.assertTrue(len(Utility.OS_NAME)) self.assertTrue(Utility.PAGE_SIZE > 0) self.assertEqual(-2 ** 31, Utility.INT32MIN) self.assertEqual(2 ** 31 - 1, Utility.INT32MAX) self.assertEqual(2 ** 32 - 1, Utility.UINT32MAX) self.assertEqual(-2 ** 63, Utility.INT64MIN) self.assertEqual(2 ** 63 - 1, Utility.INT64MAX) self.assertEqual(2 ** 64 - 1, Utility.UINT64MAX) if Utility.OS_NAME == "Linux": self.assertTrue(Utility.GetMemoryCapacity() > 0) self.assertTrue(Utility.GetMemoryUsage() > 0) self.assertTrue(3042090208, Utility.PrimaryHash("abc", (1 << 32) - 1)) self.assertTrue(16973900370012003622, Utility.PrimaryHash("abc")) self.assertTrue(702176507, Utility.SecondaryHash("abc", (1 << 32) - 1)) self.assertTrue(1765794342254572867, Utility.SecondaryHash("abc")) self.assertEqual(0, Utility.EditDistanceLev("", "")) self.assertEqual(1, Utility.EditDistanceLev("ac", "abc")) self.assertEqual(1, Utility.EditDistanceLev("あいう", "あう")) int_seq = Utility.SerializeInt(-123456) self.assertEqual(8, len(int_seq)) self.assertEqual(-123456, Utility.DeserializeInt(int_seq)) float_seq = Utility.SerializeFloat(-123.456) self.assertEqual(8, len(float_seq)) self.assertEqual(-123.456, Utility.DeserializeFloat(float_seq)) # Status tests. def testStatus(self): status = Status() self.assertEqual(Status.SUCCESS, status) self.assertEqual(Status.SUCCESS, status.GetCode()) self.assertEqual("", status.GetMessage()) self.assertTrue(status.IsOK()) status.Set(Status.NOT_FOUND_ERROR, "foobar") self.assertEqual(Status.NOT_FOUND_ERROR, status) self.assertEqual("NOT_FOUND_ERROR: foobar", str(status)) self.assertTrue("foobar" in repr(status)) self.assertFalse(status.IsOK()) s2 = Status(Status.NOT_IMPLEMENTED_ERROR, "void") status.Join(s2) self.assertEqual("NOT_FOUND_ERROR: foobar", str(status)) status.Set(Status.SUCCESS, "OK") status.Join(s2) self.assertEqual("NOT_IMPLEMENTED_ERROR: void", str(status)) try: status.OrDie() except StatusException as e: self.assertTrue("void" in str(e)) else: self.fail("no exception") self.assertEqual("SUCCESS", Status.CodeName(Status.SUCCESS)) self.assertEqual("INFEASIBLE_ERROR", Status.CodeName(Status.INFEASIBLE_ERROR)) # Basic tests. def testBasic(self): confs = [ {"path": "casket.tkh", "open_params": {"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100, "fbp_capacity": 64, "concurrent": True}, "rebuild_params": {"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100, "fbp_capacity": 64}, "synchronize_params": {}, "expected_class": "HashDBM"}, {"path": "casket.tkt", "open_params": {"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100, "fbp_capacity": 64, "max_page_size": 100, "max_branches": 4, "max_cached_pages": 1, "key_comparator": "decimal", "concurrent": True}, "rebuild_params": {"update_mode": "UPDATE_APPENDING", "offset_width": 3, "align_pow": 1, "num_buckets": 100, "fbp_capacity": 64, "max_page_size": 100, "max_branches": 4, "max_cached_pages": 1}, "synchronize_params": {}, "expected_class": "TreeDBM"}, {"path": "casket.tks", "open_params": {"offset_width": 3, "step_unit": 2, "max_level": 3, "sort_mem_size": 100, "insert_in_order": False, "max_cached_records": 8, "concurrent": True}, "rebuild_params": {"offset_width": 3, "step_unit": 2, "max_level": 3, "sort_mem_size": 100, "max_cached_records": 8}, "synchronize_params": {"reducer": "last"}, "expected_class": "SkipDBM"}, {"path": "casket.tiny", "open_params": {"num_buckets": 10}, "rebuild_params": {"num_buckets": 10}, "synchronize_params": {}, "expected_class": "TinyDBM"}, {"path": "casket.baby", "open_params": {"key_comparator": "decimal"}, "rebuild_params": {}, "synchronize_params": {}, "expected_class": "BabyDBM"}, {"path": "casket.cache", "open_params": {"cap_rec_num": 10000, "cap_mem_size": 1000000}, "rebuild_params": {"cap_rec_num": 10000}, "synchronize_params": {}, "expected_class": "CacheDBM"}, {"path": "casket.stdhash", "open_params": {"num_buckets": 10}, "rebuild_params": {}, "synchronize_params": {}, "expected_class": "StdHashDBM"}, {"path": "casket.stdtree", "open_params": {}, "rebuild_params": {}, "synchronize_params": {}, "expected_class": "StdTreeDBM"}, {"path": "casket", "open_params": {"num_shards": 4, "dbm": "hash", "num_buckets": 100}, "rebuild_params": {}, "synchronize_params": {}, "expected_class": "HashDBM"}, ] for conf in confs: path = conf["path"] path = self._make_tmp_path(path) if path else "" dbm = DBM() self.assertEqual(0, len(dbm)) open_params = conf["open_params"].copy() open_params["truncate"] = True self.assertFalse(dbm.IsOpen()) self.assertEqual(Status.SUCCESS, dbm.Open(path, True, **open_params)) self.assertTrue(dbm.IsOpen()) inspect = dbm.Inspect() class_name = inspect["class"] self.assertEqual(conf["expected_class"], class_name) for i in range(0, 20): key = "{:08d}".format(i) value = "{:d}".format(i) self.assertEqual(Status.SUCCESS, dbm.Set(key, value, False)) for i in range(0, 20, 2): key = "{:08d}".format(i) self.assertEqual(Status.SUCCESS, dbm.Remove(key)) if class_name in ("HashDBM", "TreeDBM", "TinyDBM", "BabyDBM"): self.assertEqual(Status.SUCCESS, dbm.Set("日本", "東京")) self.assertEqual("東京", dbm.GetStr("日本")) self.assertEqual(Status.SUCCESS, dbm.Remove("日本")) self.assertEqual(Status.SUCCESS, dbm.Synchronize(False, **conf["synchronize_params"])) self.assertEqual(10, dbm.Count()) self.assertTrue(dbm.GetFileSize() > 0) if path: self.assertTrue(path in dbm.GetFilePath()) self.assertTrue(dbm.GetTimestamp() > 0) self.assertTrue(dbm.IsWritable()) self.assertTrue(dbm.IsHealthy()) if class_name in ("TreeDBM", "SkipDBM", "BabyDBM", "StdTreeDBM"): self.assertTrue(dbm.IsOrdered()) else: self.assertFalse(dbm.IsOrdered()) self.assertEqual(10, len(dbm)) self.assertTrue(conf["expected_class"] in repr(dbm)) self.assertTrue(conf["expected_class"] in str(dbm)) for i in range(0, 20): key = "{:08d}".format(i) value = "new-{:d}".format(i) status = dbm.Set(key, value, False) if i % 2 == 0: self.assertTrue(status == Status.SUCCESS) else: self.assertTrue(status == Status.DUPLICATION_ERROR) sv = dbm.SetAndGet("98765", "apple", False) self.assertEqual(Status.SUCCESS, sv[0]) self.assertEqual(None, sv[1]) if class_name in ("TreeDBM", "TreeDBM", "TinyDBM", "BabyDBM"): sv = dbm.SetAndGet("98765", "orange", False) self.assertEqual(Status.DUPLICATION_ERROR, sv[0]) self.assertEqual("apple", sv[1]) sv = dbm.SetAndGet("98765", b"orange", True) self.assertEqual(Status.SUCCESS, sv[0]) self.assertEqual(b"apple", sv[1]) self.assertEqual("orange", dbm.GetStr("98765")) sv = dbm.RemoveAndGet("98765") self.assertEqual(Status.SUCCESS, sv[0]) self.assertEqual("orange", sv[1]) sv = dbm.RemoveAndGet("98765") self.assertEqual(Status.NOT_FOUND_ERROR, sv[0]) self.assertEqual(None, sv[1]) self.assertEqual(Status.SUCCESS, dbm.Set("98765", "banana")) self.assertEqual(Status.SUCCESS, dbm.Remove("98765")) self.assertEqual(Status.SUCCESS, dbm.Synchronize(False, **conf["synchronize_params"])) records = {} for i in range(0, 20): key = "{:08d}".format(i) value = "new-{:d}".format(i) if i % 2 == 0 else "{:d}".format(i) self.assertEqual(value.encode(), dbm.Get(key.encode())) self.assertEqual(value.encode(), dbm.Get(key)) self.assertEqual(value, dbm.GetStr(key.encode())) self.assertEqual(value, dbm.GetStr(key)) status = Status() rec_value = dbm.Get(key.encode(), status) self.assertEqual(value.encode(), rec_value) self.assertEqual(Status.SUCCESS, status) records[key] = value self.assertEqual(Status.SUCCESS, dbm.Rebuild(**conf["rebuild_params"])) it_records = {} iter = dbm.MakeIterator() status = Status() record = iter.Get(status) self.assertEqual(Status.NOT_FOUND_ERROR, status) self.assertEqual(None, record) self.assertEqual(Status.SUCCESS, iter.First()) self.assertTrue("0000" in repr(iter)) self.assertTrue("0000" in str(iter)) while True: status = Status() record = iter.Get(status) if status != Status.SUCCESS: self.assertEqual(Status.NOT_FOUND_ERROR, status) break self.assertEqual(2, len(record)) it_records[record[0].decode()] = record[1].decode() record_str = iter.GetStr(status) self.assertEqual(Status.SUCCESS, status) self.assertEqual(2, len(record_str)) self.assertEqual(record[0].decode(), record_str[0]) self.assertEqual(record[1].decode(), record_str[1]) self.assertEqual(record_str[0], iter.GetKeyStr()) self.assertEqual(record_str[1], iter.GetValueStr()) self.assertEqual(Status.SUCCESS, iter.Next()) self.assertEqual(records, it_records) it_records = {} for key, value in dbm: it_records[key.decode()] = value.decode() self.assertEqual(records, it_records) self.assertEqual(Status.SUCCESS, iter.Jump("00000011")) self.assertEqual("00000011", iter.GetKey().decode()) self.assertEqual("11", iter.GetValue().decode()) if dbm.IsOrdered(): self.assertEqual(Status.SUCCESS, iter.Last()) it_records = {} while True: status = Status() record = iter.Get(status) if status != Status.SUCCESS: self.assertEqual(Status.NOT_FOUND_ERROR, status) break self.assertEqual(2, len(record)) it_records[record[0].decode()] = record[1].decode() self.assertEqual(Status.SUCCESS, iter.Previous()) self.assertEqual(records, it_records) if path: base, ext = os.path.splitext(path) copy_path = base + "-copy" + ext self.assertEqual(Status.SUCCESS, dbm.CopyFileData(copy_path)) copy_dbm = DBM() if "." in path: self.assertEqual(Status.SUCCESS, copy_dbm.Open(copy_path, False)) else: params = {"dbm": conf["expected_class"]} if "num_shards" in open_params: params["num_shards"] = 0 self.assertEqual(Status.SUCCESS, copy_dbm.Open(copy_path, False, **params)) self.assertTrue(copy_dbm.IsHealthy()) it_records = {} for key, value in copy_dbm: it_records[key.decode()] = value.decode() self.assertEqual(records, it_records) self.assertEqual(Status.SUCCESS, copy_dbm.Close()) if class_name in ("HashDBM", "TreeDBM"): restored_path = copy_path + "-restored" self.assertEqual(Status.SUCCESS, DBM.RestoreDatabase( copy_path, restored_path, class_name, -1)) export_dbm = DBM() self.assertEqual(Status.SUCCESS, export_dbm.Open("", True, dbm="BabyDBM")) self.assertEqual(Status.SUCCESS, dbm.Export(export_dbm)) it_records = {} for key, value in export_dbm: it_records[key.decode()] = value.decode() self.assertEqual(records, it_records) self.assertEqual(Status.SUCCESS, export_dbm.Clear()) self.assertEqual(0, len(export_dbm)) self.assertEqual(Status.SUCCESS, export_dbm.Set("1", "100")) value = export_dbm.Increment("10000", 2, 10000, status) self.assertEqual(Status.SUCCESS, status) self.assertEqual(value, 10002) value = export_dbm.Increment("10000", Utility.INT64MIN, 0, status) self.assertEqual(Status.SUCCESS, status) self.assertEqual(value, 10002) self.assertEqual(Status.DUPLICATION_ERROR, export_dbm.Set("1", "101", False)) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", "100", "101")) value = export_dbm.Increment("10000", 2) self.assertEqual(value, 10004) self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchange("1", "100", "101")) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", "101", None)) value = export_dbm.Get("1", status) self.assertEqual(Status.NOT_FOUND_ERROR, status) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", None, "zzz")) self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchange("1", None, "yyy")) self.assertEqual("zzz", export_dbm.GetStr("1", status)) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("1", "zzz", None)) self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchange( "xyz", DBM.ANY_DATA, DBM.ANY_DATA)) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange("xyz", None, "abc")) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange( "xyz", DBM.ANY_DATA, DBM.ANY_DATA)) self.assertEqual("abc", export_dbm.GetStr("xyz", status)) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange( "xyz", DBM.ANY_DATA, "def")) self.assertEqual("def", export_dbm.GetStr("xyz", status)) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchange( "xyz", DBM.ANY_DATA, None)) self.assertTrue(export_dbm.GetStr("xyz", status) == None) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti( (("hop", None), ("step", None)), (("hop", "one"), ("step", "two")))) self.assertEqual("one", export_dbm.GetStr("hop")) self.assertEqual("two", export_dbm.GetStr("step")) status, value = export_dbm.CompareExchangeAndGet("xyz", None, "123"); self.assertEqual(Status.SUCCESS, status) self.assertEqual(None, value) status, value = export_dbm.CompareExchangeAndGet("xyz", "123", DBM.ANY_DATA); self.assertEqual(Status.SUCCESS, status) self.assertEqual("123", value) status, value = export_dbm.CompareExchangeAndGet("xyz", DBM.ANY_DATA, None); self.assertEqual(Status.SUCCESS, status) self.assertEqual(b"123", value) self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchangeMulti( (("hop", "one"), ("step", None)), (("hop", "uno"), ("step", "dos")))) self.assertEqual("one", export_dbm.GetStr("hop")) self.assertEqual("two", export_dbm.GetStr("step")) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti( (("hop", "one"), ("step", "two")), (("hop", "1"), ("step", "2")))) self.assertEqual("1", export_dbm.GetStr("hop")) self.assertEqual("2", export_dbm.GetStr("step")) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti( (("hop", "1"), ("step", "2")), (("hop", None), ("step", None)))) self.assertEqual(None, export_dbm.GetStr("hop")) self.assertEqual(None, export_dbm.GetStr("step")) self.assertEqual(Status.INFEASIBLE_ERROR, export_dbm.CompareExchangeMulti( [("xyz", DBM.ANY_DATA)], [("xyz", "abc")])) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti( [("xyz", None)], [("xyz", "abc")])) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti( [("xyz", DBM.ANY_DATA)], [("xyz", "def")])) self.assertEqual("def", export_dbm.GetStr("xyz")) self.assertEqual(Status.SUCCESS, export_dbm.CompareExchangeMulti( [("xyz", DBM.ANY_DATA)], [("xyz", None)])) self.assertEqual(None, export_dbm.GetStr("xyz")) export_iter = export_dbm.MakeIterator() self.assertEqual(Status.SUCCESS, export_iter.First()) self.assertEqual(Status.SUCCESS, export_iter.Set("foobar")) self.assertEqual(Status.SUCCESS, export_iter.Remove()) self.assertEqual(0, len(export_dbm)) self.assertEqual(Status.SUCCESS, export_dbm.Append("foo", "bar", ",")) self.assertEqual(Status.SUCCESS, export_dbm.Append("foo", "baz", ",")) self.assertEqual(Status.SUCCESS, export_dbm.Append("foo", "qux", "")) self.assertEqual("bar,bazqux", export_dbm.GetStr("foo")) export_dbm["abc"] = "defg" self.assertEqual("defg", export_dbm["abc"]) self.assertTrue("abc" in export_dbm) del export_dbm["abc"] try: export_dbm["abc"] except StatusException as e: self.assertEqual(Status.NOT_FOUND_ERROR, e.GetStatus()) self.assertFalse("abc" in export_dbm) self.assertEqual(Status.SUCCESS, export_dbm.SetMulti(True, one="first", two="second")) self.assertEqual(Status.SUCCESS, export_dbm.AppendMulti(":", one="1", two="2")) ret_records = export_dbm.GetMulti("one", "two", "three") self.assertEqual("first:1".encode(), ret_records.get("one".encode())) self.assertEqual("second:2".encode(), ret_records.get("two".encode())) self.assertEqual(None, ret_records.get("third".encode())) ret_records = export_dbm.GetMultiStr("one", "two", "three") self.assertEqual("first:1", ret_records.get("one")) self.assertEqual("second:2", ret_records.get("two")) self.assertEqual(None, ret_records.get("third")) self.assertEqual(Status.SUCCESS, export_dbm.RemoveMulti("one", "two")) self.assertEqual(Status.NOT_FOUND_ERROR, export_dbm.RemoveMulti("two", "three")) status = Status() self.assertEqual(None, export_dbm.Get("one", status)) self.assertEqual(Status.NOT_FOUND_ERROR, status) status = Status() self.assertEqual(None, export_dbm.Get("two", status)) self.assertEqual(Status.NOT_FOUND_ERROR, status) status = Status() self.assertEqual(None, export_dbm.Get("three", status)) self.assertEqual(Status.NOT_FOUND_ERROR, status) self.assertEqual(Status.SUCCESS, export_dbm.Set("zero", "foo")) self.assertEqual(Status.SUCCESS, export_dbm.Rekey("zero", "one", True)) self.assertEqual(None, export_dbm.Get("zero")) self.assertEqual("foo", export_dbm.GetStr("one")) step_count = 0 self.assertEqual(Status.SUCCESS, export_iter.First()) while True: status.Set(Status.UNKNOWN_ERROR) record = export_iter.Step(status) if not record: self.assertEqual(Status.NOT_FOUND_ERROR, status) break self.assertEqual(Status.SUCCESS, status) step_count += 1 self.assertEqual(export_dbm.Count(), step_count) step_count = 0 self.assertEqual(Status.SUCCESS, export_iter.First()) while True: status.Set(Status.UNKNOWN_ERROR) record = export_iter.StepStr(status) if not record: self.assertEqual(Status.NOT_FOUND_ERROR, status) break self.assertEqual(Status.SUCCESS, status) step_count += 1 self.assertEqual(export_dbm.Count(), step_count) pop_count = 0 while True: status.Set(Status.UNKNOWN_ERROR) record = export_dbm.PopFirst(status) if not record: self.assertEqual(Status.NOT_FOUND_ERROR, status) break self.assertEqual(Status.SUCCESS, status) pop_count += 1 self.assertEqual(step_count, pop_count) self.assertEqual(0, export_dbm.Count()) self.assertEqual(Status.SUCCESS, export_dbm.SetMulti( False, japan="tokyo", china="beijing", korea="seoul", france="paris")) pop_count = 0 while True: status.Set(Status.UNKNOWN_ERROR) record = export_dbm.PopFirstStr(status) if not record: self.assertEqual(Status.NOT_FOUND_ERROR, status) break self.assertEqual(Status.SUCCESS, status) pop_count += 1 self.assertEqual(4, pop_count) self.assertEqual(Status.SUCCESS, export_dbm.PushLast("foo", 0)) record = export_dbm.PopFirst() self.assertEqual(record[0], b"\0\0\0\0\0\0\0\0") self.assertEqual(record[1], b"foo") self.assertEqual(Status.SUCCESS, export_dbm.Close()) self.assertEqual(Status.SUCCESS, dbm.Close()) if path: self.assertEqual(Status.SUCCESS, dbm.Open(path, False, **open_params)) self.assertEqual(Status.SUCCESS, dbm.Close()) # Basic process-related functions. def testProcess(self): path = self._make_tmp_path("casket.tkh") dbm = DBM() self.assertEqual(Status.SUCCESS, dbm.Open(path, True, truncate=True, num_buckets=1000)) self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: None, True)) self.assertEqual(None, dbm.GetStr("abc")) self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: False, True)) self.assertEqual(None, dbm.GetStr("abc")) self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: "ABCDE", True)) self.assertEqual("ABCDE", dbm.GetStr("abc")) def Processor1(k, v): self.assertEqual(b"abc", k) self.assertEqual(b"ABCDE", v) return None self.assertEqual(Status.SUCCESS, dbm.Process("abc", Processor1, False)) self.assertEqual(Status.SUCCESS, dbm.Process("abc", lambda k, v: False, True)) self.assertEqual(None, dbm.GetStr("abc")) def Processor2(k, v): self.assertEqual(b"abc", k) self.assertEqual(None, v) return None self.assertEqual(Status.SUCCESS, dbm.Process("abc", Processor2, False)) for i in range(10): self.assertEqual(Status.SUCCESS, dbm.Process(i, lambda k, v: i * i, True)) self.assertEqual(10, dbm.Count()) counters = {"empty": 0, "full": 0} def Processor3(k, v): if k: counters["full"] += 1 self.assertEqual(int(k) ** 2, int(v)) else: counters["empty"] += 1 return None self.assertEqual(Status.SUCCESS, dbm.ProcessEach(Processor3, False)) self.assertEqual(2, counters["empty"]) self.assertEqual(10, counters["full"]) def Processor4(k, v): if not k: return return int(int(v) ** 0.5) self.assertEqual(Status.SUCCESS, dbm.ProcessEach(Processor4, True)) def Processor5(k, v): if not k: return self.assertEqual(int(k), int(v)) return False self.assertEqual(Status.SUCCESS, dbm.ProcessEach(Processor5, True)) self.assertEqual(0, dbm.Count()) ops = [ ("one", lambda key, value: "hop"), ("two", lambda key, value: "step"), ("three", lambda key, value: "jump"), ] self.assertEqual(Status.SUCCESS, dbm.ProcessMulti(ops, True)) ops = [ ("one", lambda key, value: False), ("two", lambda key, value: False), ("three", lambda key, value: value.decode() * 2 if value else "x"), ("four", lambda key, value: value.decode() * 2 if value else "x"), ("three", lambda key, value: value.decode() * 2 if value else "x"), ("four", lambda key, value: value.decode() * 2 if value else "x"), ] self.assertEqual(Status.SUCCESS, dbm.ProcessMulti(ops, True)) self.assertEqual(2, dbm.Count()) self.assertEqual(None, dbm.GetStr("one")) self.assertEqual(None, dbm.GetStr("two")) self.assertEqual("jumpjumpjumpjump", dbm.GetStr("three")) self.assertEqual("xx", dbm.GetStr("four")) self.assertEqual(Status.SUCCESS, dbm.Close()) # Thread tests. def testThread(self): dbm = DBM() self.assertEqual(Status.SUCCESS, dbm.Open( "casket.tkh", True, truncate=True, num_buckets=1000)) rnd_state = random.Random() num_records = 5000 num_threads = 5 records = {} class Task(threading.Thread): def __init__(self, test, thid): threading.Thread.__init__(self) self.thid = thid self.test = test def run(self): for i in range(0, num_records): key_num = rnd_state.randint(1, num_records) key_num = key_num - key_num % num_threads + self.thid key = str(key_num) value = str(key_num * key_num) if rnd_state.randint(0, num_records) == 0: self.test.assertEqual(Status.SUCCESS, dbm.Rebuild()) elif rnd_state.randint(0, 10) == 0: iter = dbm.MakeIterator() iter.Jump(key) status = Status() record = iter.Get(status) if status == Status.SUCCESS: self.test.assertEqual(2, len(record)) self.test.assertEqual(key, record[0].decode()) self.test.assertEqual(value, record[1].decode()) status = iter.Next() self.test.assertTrue(status == Status.SUCCESS or status == Status.NOT_FOUND_ERROR) elif rnd_state.randint(0, 4) == 0: status = Status() rec_value = dbm.Get(key, status) if status == Status.SUCCESS: self.test.assertEqual(value, rec_value.decode()) else: self.test.assertEqual(Status.NOT_FOUND_ERROR, status) elif rnd_state.randint(0, 4) == 0: status = dbm.Remove(key) if status == Status.SUCCESS: del records[key] else: self.test.assertEqual(Status.NOT_FOUND_ERROR, status) else: overwrite = rnd_state.randint(0, 2) == 0 status = dbm.Set(key, value, overwrite) if status == Status.SUCCESS: records[key] = value else: self.test.assertEqual(Status.DUPLICATION_ERROR, status) if rnd_state.randint(0, 10) == 0: time.sleep(0.00001) threads = [] for thid in range(0, num_threads): threads.append(Task(self, thid)) for th in threads: th.start() for th in threads: th.join() it_records = {} for key, value in dbm: it_records[key.decode()] = value.decode() self.assertEqual(records, it_records) self.assertEqual(Status.SUCCESS, dbm.Close()) # Search tests. def testSearch(self): confs = [ {"path": "casket.tkh", "open_params": {"num_buckets": 100}}, {"path": "casket.tkt", "open_params": {"num_buckets": 100}}, {"path": "casket.tks", "open_params": {"max_level": 8}}, {"path": "", "open_params": {"dbm": "TinyDBM", "num_buckets": 100}}, {"path": "", "open_params": {"dbm": "BabyDBM"}}, ] for conf in confs: path = conf["path"] path = self._make_tmp_path(path) if path else "" dbm = DBM() open_params = conf["open_params"].copy() open_params["truncate"] = True self.assertEqual(Status.SUCCESS, dbm.Open(path, True, **open_params)) for i in range(1, 101): key = "{:08d}".format(i) value = "{:d}".format(i) self.assertEqual(Status.SUCCESS, dbm.Set(key, value, False)) self.assertEqual(Status.SUCCESS, dbm.Synchronize(False)) self.assertEqual(100, dbm.Count()) self.assertEqual(12, len(dbm.Search("contain", "001"))) self.assertEqual(3, len(dbm.Search("contain", "001", 3))) self.assertEqual(10, len(dbm.Search("begin", "0000001"))) self.assertEqual(10, len(dbm.Search("end", "1"))) self.assertEqual(10, len(dbm.Search("regex", r"^\d+1$"))) self.assertEqual(10, len(dbm.Search("regex", r"^\d+1$", 0,))) self.assertEqual(3, len(dbm.Search("edit", "00000100", 3))) self.assertEqual(3, len(dbm.Search("editbin", "00000100", 3))) with self.assertRaises(StatusException): self.assertRaises(dbm.Search("foo", "00000100", 3)) self.assertEqual(Status.SUCCESS, dbm.Close()) # Export tests. def testExport(self): dbm = DBM() dest_path = self._make_tmp_path("casket.txt") self.assertEqual(Status.SUCCESS, dbm.Open("", True)) for i in range(1, 101): key = "{:08d}".format(i) value = "{:d}".format(i) self.assertEqual(Status.SUCCESS, dbm.Set(key, value, False)) file = File() self.assertEqual(Status.SUCCESS, file.Open(dest_path, True, truncate=True)) self.assertEqual(Status.SUCCESS, dbm.ExportToFlatRecords(file)) self.assertEqual(Status.SUCCESS, dbm.Clear()) self.assertEqual(0, dbm.Count()) self.assertEqual(Status.SUCCESS, dbm.ImportFromFlatRecords(file)) self.assertEqual(100, dbm.Count()) self.assertEqual(Status.SUCCESS, file.Close()) file = File() self.assertEqual(Status.SUCCESS, file.Open(dest_path, True, truncate=True)) self.assertEqual(Status.SUCCESS, dbm.ExportKeysAsLines(file)) self.assertEqual(Status.SUCCESS, file.Close()) self.assertEqual(Status.SUCCESS, dbm.Close()) file = File() self.assertEqual(Status.SUCCESS, file.Open(dest_path, False)) self.assertTrue("File" in repr(file)) self.assertTrue("File" in str(file)) self.assertEqual(12, len(file.Search("contain", "001"))) self.assertEqual(3, len(file.Search("contain", "001", 3))) self.assertEqual(10, len(file.Search("begin", "0000001"))) self.assertEqual(10, len(file.Search("end", "1"))) self.assertEqual(10, len(file.Search("regex", r"^\d+1$"))) self.assertEqual(10, len(file.Search("regex", r"^\d+1$", 0))) self.assertEqual(3, len(file.Search("edit", "00000100", 3))) self.assertEqual(3, len(file.Search("editbin", "00000100", 3))) with self.assertRaises(StatusException): self.assertRaises(file.Search("foo", "00000100", 3)) self.assertEqual(Status.SUCCESS, file.Close()) # AsyncDBM tests. def testAsyncDBM(self): dbm = DBM() path = self._make_tmp_path("casket.tkh") copy_path = self._make_tmp_path("casket-copy.tkh") self.assertEqual(Status.SUCCESS, dbm.Open(path, True, num_buckets=100, concurrent=True)) adbm = AsyncDBM(dbm, 4) self.assertTrue("AsyncDBM" in repr(adbm)) self.assertTrue("AsyncDBM" in str(adbm)) set_future = adbm.Set("one", "hop") self.assertTrue("Future" in repr(set_future)) self.assertTrue("Future" in str(set_future)) set_future.Wait(0) self.assertTrue(set_future.Wait()) self.assertEqual(Status.SUCCESS, set_future.Get()) self.assertEqual(Status.DUPLICATION_ERROR, adbm.Set("one", "more", False).Get()) self.assertEqual(Status.SUCCESS, adbm.Set("two", "step", False).Get()) self.assertEqual(Status.SUCCESS, adbm.Set("three", "jump", False).Get()) self.assertEqual(Status.SUCCESS, adbm.Append("three", "go", ":").Get()) get_result = adbm.Get("one").Get() self.assertEqual(Status.SUCCESS, get_result[0]) self.assertEqual(b"hop", get_result[1]) self.assertEqual("step", adbm.GetStr("two").Get()[1]) self.assertEqual("jump:go", adbm.GetStr("three").Get()[1]) self.assertEqual(Status.SUCCESS, adbm.Get("three").Get()[0]) self.assertEqual(Status.SUCCESS, adbm.Remove("one").Get()) self.assertEqual(Status.NOT_FOUND_ERROR, adbm.Remove("one").Get()) self.assertEqual(Status.SUCCESS, adbm.Remove("two").Get()) self.assertEqual(Status.SUCCESS, adbm.Remove("three").Get()) self.assertEqual(0, dbm.Count()) set_future = adbm.SetMulti(False, one="hop", two="step", three="jump") self.assertEqual(Status.SUCCESS, set_future.Get()) self.assertEqual(Status.SUCCESS, adbm.AppendMulti(":", three="go").Get()) get_result = adbm.GetMulti("one", "two").Get() self.assertEqual(Status.SUCCESS, get_result[0]) self.assertEqual(b"hop", get_result[1][b"one"]) self.assertEqual(b"step", get_result[1][b"two"]) get_result = adbm.GetMultiStr("one", "two", "three", "hoge").Get() self.assertEqual(Status.NOT_FOUND_ERROR, get_result[0]) self.assertEqual("hop", get_result[1]["one"]) self.assertEqual("step", get_result[1]["two"]) self.assertEqual("jump:go", get_result[1]["three"]) self.assertEqual(Status.SUCCESS, adbm.RemoveMulti("one", "two", "three").Get()) self.assertEqual(0, dbm.Count()) incr_result = adbm.Increment("num", 5, 100).Get() self.assertEqual(Status.SUCCESS, incr_result[0]) self.assertEqual(105, incr_result[1]) self.assertEqual(110, adbm.Increment("num", 5, 100).Get()[1]) self.assertEqual(Status.SUCCESS, adbm.Remove("num").Get()) self.assertEqual(Status.SUCCESS, adbm.CompareExchange("one", None, "ichi").Get()) self.assertEqual("ichi", adbm.GetStr("one").Get()[1]) self.assertEqual(Status.SUCCESS, adbm.CompareExchange("one", "ichi", "ni").Get()) self.assertEqual("ni", adbm.GetStr("one").Get()[1]) self.assertEqual(Status.INFEASIBLE_ERROR, adbm.CompareExchange( "xyz", DBM.ANY_DATA, DBM.ANY_DATA).Get()) self.assertEqual(Status.SUCCESS, adbm.CompareExchange( "xyz", None, "abc").Get()) self.assertEqual(Status.SUCCESS, adbm.CompareExchange( "xyz", DBM.ANY_DATA, DBM.ANY_DATA).Get()) self.assertEqual("abc", adbm.GetStr("xyz").Get()[1]) self.assertEqual(Status.SUCCESS, adbm.CompareExchange("xyz", DBM.ANY_DATA, "def").Get()) self.assertEqual("def", adbm.GetStr("xyz").Get()[1]) self.assertEqual(Status.SUCCESS, adbm.CompareExchange("xyz", DBM.ANY_DATA, None).Get()) self.assertEqual(Status.NOT_FOUND_ERROR, adbm.GetStr("xyz").Get()[0]) self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti( [("one", "ni"), ("two", None)], [("one", "san"), ("two", "uno")]).Get()) self.assertEqual("san", adbm.GetStr("one").Get()[1]) self.assertEqual("uno", adbm.GetStr("two").Get()[1]) self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti( [("one", "san"), ("two", "uno")], [("one", None), ("two", None)]).Get()) self.assertEqual(Status.INFEASIBLE_ERROR, adbm.CompareExchangeMulti( [("xyz", DBM.ANY_DATA)], [("xyz", "abc")]).Get()) self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti( [("xyz", None)], [("xyz", "abc")]).Get()) self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti( [("xyz", DBM.ANY_DATA)], [("xyz", "abc")]).Get()) self.assertEqual("abc", adbm.GetStr("xyz").Get()[1]) self.assertEqual(Status.SUCCESS, adbm.CompareExchangeMulti( [("xyz", DBM.ANY_DATA)], [("xyz", None)]).Get()) self.assertEqual(Status.NOT_FOUND_ERROR, adbm.GetStr("xyz").Get()[0]) self.assertEqual(0, dbm.Count()) self.assertEqual(Status.SUCCESS, adbm.Set("hello", "world", False).Get()) self.assertEqual(Status.SUCCESS, adbm.Synchronize(False).Get()) self.assertEqual(Status.SUCCESS, adbm.Rebuild().Get()) self.assertEqual(1, dbm.Count()) self.assertEqual(Status.SUCCESS, adbm.CopyFileData(copy_path).Get()) copy_dbm = DBM() self.assertEqual(Status.SUCCESS, copy_dbm.Open(copy_path, True)) self.assertEqual(1, copy_dbm.Count()) self.assertEqual(Status.SUCCESS, copy_dbm.Clear()) self.assertEqual(0, copy_dbm.Count()) self.assertEqual(Status.SUCCESS, adbm.Export(copy_dbm).Get()) self.assertEqual(1, copy_dbm.Count()) self.assertEqual(Status.SUCCESS, copy_dbm.Close()) os.remove(copy_path) copy_file = File() self.assertEqual(Status.SUCCESS, copy_file.Open(copy_path, True)) self.assertEqual(Status.SUCCESS, adbm.ExportToFlatRecords(copy_file).Get()) self.assertEqual(Status.SUCCESS, adbm.Clear().Get()) self.assertEqual(0, dbm.Count()) self.assertEqual(Status.SUCCESS, adbm.ImportFromFlatRecords(copy_file).Get()) self.assertEqual(1, dbm.Count()) self.assertEqual(Status.SUCCESS, copy_file.Close()) async def async_main(): await adbm.Set("hello", "good-bye", True) await adbm.Set("hi", "bye", True) await adbm.Set("chao", "adios", True) asyncio.run(async_main()) self.assertEqual("good-bye", dbm.GetStr("hello")) search_result = adbm.Search("begin", "h").Get() self.assertEqual(Status.SUCCESS, search_result[0]) self.assertEqual(2, len(search_result[1])) self.assertTrue("hello" in search_result[1]) self.assertTrue("hi" in search_result[1]) self.assertEqual(Status.SUCCESS, adbm.Clear().Get()) self.assertEqual(Status.SUCCESS, adbm.Set("aa", "AAA").Get()) self.assertEqual(Status.SUCCESS, adbm.Rekey("aa", "bb").Get()) get_result = adbm.GetStr("bb").Get() self.assertEqual(Status.SUCCESS, get_result[0]) self.assertEqual("AAA", get_result[1]) pop_result = adbm.PopFirst().Get() self.assertEqual(Status.SUCCESS, pop_result[0]) self.assertEqual(b"bb", pop_result[1]) self.assertEqual(b"AAA", pop_result[2]) self.assertEqual(Status.SUCCESS, adbm.Set("cc", "CCC").Get()) pop_result = adbm.PopFirstStr().Get() self.assertEqual(Status.SUCCESS, pop_result[0]) self.assertEqual("cc", pop_result[1]) self.assertEqual("CCC", pop_result[2]) self.assertEqual(Status.SUCCESS, adbm.PushLast("foo", 0).Get()) pop_result = adbm.PopFirst().Get() self.assertEqual(Status.SUCCESS, pop_result[0]) self.assertEqual(b"\0\0\0\0\0\0\0\0", pop_result[1]) self.assertEqual(b"foo", pop_result[2]) adbm.Destruct() self.assertEqual(Status.SUCCESS, dbm.Close()) # File tests. def testFile(self): path = self._make_tmp_path("casket.txt") file = File() self.assertEqual(Status.SUCCESS, file.Open( path, True, truncate=True, file="pos-atom", block_size=512, access_options="padding:pagecache")) self.assertEqual(Status.SUCCESS, file.Write(5, "12345")) self.assertEqual(Status.SUCCESS, file.Write(0, "ABCDE")) self.assertEqual(10, file.Append("FGH")) self.assertEqual(13, file.Append("IJ")) self.assertEqual(15, file.GetSize()) self.assertTrue(file.GetPath().find("casket.txt") > 0) self.assertEqual(Status.SUCCESS, file.Synchronize(False)) self.assertEqual(Status.SUCCESS, file.Truncate(12)) self.assertEqual(12, file.GetSize()) self.assertEqual(b"ABCDE12345FG", file.Read(0, 12)) self.assertEqual("ABCDE12345FG", file.ReadStr(0, 12)) self.assertEqual(b"DE123", file.Read(3, 5)) self.assertEqual("DE123", file.ReadStr(3, 5)) status = Status() self.assertEqual(None, file.ReadStr(1024, 10, status)) self.assertEqual(Status.INFEASIBLE_ERROR, status) self.assertEqual(Status.SUCCESS, file.Close()) self.assertEqual(Status.SUCCESS, file.Open(path, False)) self.assertEqual(512, file.GetSize()) self.assertEqual("E12345F", file.ReadStr(4, 7)) self.assertEqual(Status.SUCCESS, file.Close()) # Index tests. def testIndex(self): path = self._make_tmp_path("casket.tkt") index = Index() self.assertEqual(Status.SUCCESS, index.Open( path, True, truncate=True, num_buckets=100)) self.assertTrue("Index" in repr(index)) self.assertTrue(("path=" + path) in str(index)) self.assertFalse(("single", "1") in index) self.assertFalse(("double", "11") in index) self.assertEqual(Status.SUCCESS, index.Add("single", "1")) self.assertEqual(Status.SUCCESS, index.Add("double", "11")) self.assertTrue(("single", "1") in index) self.assertTrue(("double", "11") in index) self.assertEqual(Status.SUCCESS, index.Add("single", "2")) self.assertEqual(Status.SUCCESS, index.Add("double", "22")) self.assertEqual(Status.SUCCESS, index.Add("triple", "222")) values = index.GetValues("single") self.assertEqual(2, len(values)) self.assertEqual(b"1", values[0]) self.assertEqual(b"2", values[1]) values = index.GetValuesStr("triple", 0) self.assertEqual(1, len(values)) self.assertEqual("222", values[0]) self.assertEqual(0, len(index.GetValuesStr("foo"))) self.assertEqual(Status.SUCCESS, index.Remove("single", "1")) self.assertEqual(Status.SUCCESS, index.Remove("double", "11")) self.assertEqual(Status.NOT_FOUND_ERROR, index.Remove("triple", "x")) values = index.GetValuesStr("double") self.assertEqual(1, len(values)) self.assertEqual("22", values[0]) self.assertEqual(3, index.Count()) self.assertEqual(path, index.GetFilePath()) self.assertEqual(Status.SUCCESS, index.Synchronize(False)) self.assertEqual(Status.SUCCESS, index.Rebuild()) self.assertEqual(3, len(index)) self.assertEqual(Status.SUCCESS, index.Clear()) self.assertEqual(0, index.Count()) self.assertTrue(index.IsOpen()) self.assertTrue(index.IsWritable()) self.assertEqual(Status.SUCCESS, index.Add("first", "1")) self.assertEqual(Status.SUCCESS, index.Add("second", "22")) self.assertEqual(Status.SUCCESS, index.Add("third", "333")) iter = index.MakeIterator() self.assertTrue("IndexIterator" in repr(iter)) self.assertTrue("unlocated" in str(iter)) iter.First() self.assertTrue("first" in repr(iter)) self.assertTrue("first" in str(iter)) record = iter.Get() self.assertTrue(record != None) self.assertTrue(b"first", record[0]) self.assertTrue(b"1", record[1]) iter.Next() record = iter.Get() self.assertTrue(record != None) self.assertTrue(b"second", record[0]) self.assertTrue(b"22", record[1]) iter.Next() record = iter.Get() self.assertTrue(record != None) self.assertTrue(b"third", record[0]) self.assertTrue(b"333", record[1]) iter.Next() self.assertEqual(None, iter.Get()) iter.Last() record = iter.GetStr() self.assertTrue(record != None) self.assertTrue("third", record[0]) self.assertTrue("333", record[1]) iter.Previous() record = iter.GetStr() self.assertTrue(record != None) self.assertTrue("second", record[0]) self.assertTrue("22", record[1]) iter.Previous() record = iter.GetStr() self.assertTrue(record != None) self.assertTrue("first", record[0]) self.assertTrue("11", record[1]) iter.Previous() self.assertEqual(None, iter.GetStr()) iter.Jump("second", "") record = iter.GetStr() self.assertTrue(record != None) self.assertTrue("second", record[0]) self.assertTrue("22", record[1]) records = {} for key, value in index: records[key.decode()] = value.decode() self.assertEqual(3, len(records)) self.assertEqual("1", records["first"]) self.assertEqual("22", records["second"]) self.assertEqual("333", records["third"]) self.assertEqual(Status.SUCCESS, index.Close()) # Main routine. def main(argv): test_names = argv if test_names: test_suite = unittest.TestSuite() for test_name in test_names: test_suite.addTest(TestTkrzw(test_name)) else: test_suite = unittest.TestLoader().loadTestsFromTestCase(TestTkrzw) unittest.TextTestRunner(verbosity=2).run(test_suite) return 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:])) # END OF FILE tkrzw-python-0.1.32/example3.py0000775000175000017500000001100514626642264015405 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Example for key comparators of the tree database # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import tkrzw # Opens a new database with the default key comparator (LexicalKeyComparator). dbm = tkrzw.DBM() open_params = { "truncate": True, } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a big-endian binary of an integer. # e.g: "\x00\x00\x00\x00\x00\x00\x00\x31" -> "hop" dbm.Set(tkrzw.Utility.SerializeInt(1), "hop").OrDie() dbm.Set(tkrzw.Utility.SerializeInt(256), "step").OrDie() dbm.Set(tkrzw.Utility.SerializeInt(32), "jump").OrDie() # Gets records with the key being a big-endian binary of an integer. print(dbm.GetStr(tkrzw.Utility.SerializeInt(1))) print(dbm.GetStr(tkrzw.Utility.SerializeInt(256))) print(dbm.GetStr(tkrzw.Utility.SerializeInt(32))) # Lists up all records, restoring keys into integers. iter = dbm.MakeIterator() iter.First() while True: record = iter.Get() if not record: break print(str(tkrzw.Utility.DeserializeInt(record[0])) + ": " + record[1].decode()) iter.Next() # Closes the database. dbm.Close().OrDie() # Opens a new database with the decimal integer comparator. open_params = { "truncate": True, "key_comparator": "Decimal", } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a decimal string of an integer. # e.g: "1" -> "hop" dbm.Set("1", "hop").OrDie() dbm.Set("256", "step").OrDie() dbm.Set("32", "jump").OrDie() # Gets records with the key being a decimal string of an integer. print(dbm.GetStr("1")) print(dbm.GetStr("256")) print(dbm.GetStr("32")) # Lists up all records, restoring keys into integers. iter = dbm.MakeIterator() iter.First() while True: record = iter.GetStr() if not record: break print("{:d}: {}".format(int(record[0]), record[1])) iter.Next() # Closes the database. dbm.Close().OrDie() # Opens a new database with the decimal real number comparator. open_params = { "truncate": True, "key_comparator": "RealNumber", } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a decimal string of a real number. # e.g: "1.5" -> "hop" dbm.Set("1.5", "hop").OrDie() dbm.Set("256.5", "step").OrDie() dbm.Set("32.5", "jump").OrDie() # Gets records with the key being a decimal string of a real number. print(dbm.GetStr("1.5")) print(dbm.GetStr("256.5")) print(dbm.GetStr("32.5")) # Lists up all records, restoring keys into floating-point numbers. iter = dbm.MakeIterator() iter.First() while True: record = iter.GetStr() if not record: break print("{:.3f}: {}".format(float(record[0]), record[1])) iter.Next() # Closes the database. dbm.Close().OrDie() # Opens a new database with the big-endian floating-point numbers comparator. open_params = { "truncate": True, "key_comparator": "FloatBigEndian", } status = dbm.Open("casket.tkt", True, **open_params).OrDie() # Sets records with the key being a big-endian binary of a floating-point number. # e.g: "\x3F\xF8\x00\x00\x00\x00\x00\x00" -> "hop" dbm.Set(tkrzw.Utility.SerializeFloat(1.5), "hop").OrDie() dbm.Set(tkrzw.Utility.SerializeFloat(256.5), "step").OrDie() dbm.Set(tkrzw.Utility.SerializeFloat(32.5), "jump").OrDie() # Gets records with the key being a big-endian binary of a floating-point number. print(dbm.GetStr(tkrzw.Utility.SerializeFloat(1.5))) print(dbm.GetStr(tkrzw.Utility.SerializeFloat(256.5))) print(dbm.GetStr(tkrzw.Utility.SerializeFloat(32.5))) # Lists up all records, restoring keys into floating-point numbers. iter = dbm.MakeIterator() iter.First() while True: record = iter.Get() if not record: break print("{:.3f}: {}".format(tkrzw.Utility.DeserializeFloat(record[0]), record[1].decode())) iter.Next() # Closes the database. dbm.Close().OrDie() # END OF FILE tkrzw-python-0.1.32/wicked.py0000775000175000017500000001155214626642264015144 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Wicked test cases # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import argparse import sys import os import re import random import time import threading import shutil from tkrzw import * # main routine def main(argv): ap = argparse.ArgumentParser( prog="wicked.py", description="Random operation tests.", formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--path", default="") ap.add_argument("--params", default="") ap.add_argument("--iter", type=int, default=10000) ap.add_argument("--threads", type=int, default=1) ap.add_argument("--random", action='store_true', default=False) args = ap.parse_args(argv) path = args.path open_params = {} open_params_exprs = [] for field in args.params.split(","): columns = field.split("=", 1) if len(columns) == 2: open_params[columns[0]] = columns[1] open_params_exprs.append(columns[0] + "=" + columns[1]) num_iterations = args.iter num_threads = args.threads print("path: {}".format(path)) print("params: {}".format(",".join(open_params_exprs))) print("num_iterations: {}".format(num_iterations)) print("num_threads: {}".format(num_threads)) print("") open_params["truncate"] = True start_mem_usage = Utility.GetMemoryUsage() dbm = DBM() dbm.Open(path, True, **open_params).OrDie() class Task(threading.Thread): def __init__(self, thid): threading.Thread.__init__(self) self.thid = thid def run(self): rnd_state = random.Random() for i in range(0, num_iterations): key_num = rnd_state.randint(1, num_iterations) key = "{:d}".format(key_num) value = "{:d}".format(i) if rnd_state.randint(0, num_iterations / 2) == 0: dbm.Rebuild().OrDie() elif rnd_state.randint(0, num_iterations / 2) == 0: dbm.Clear().OrDie() elif rnd_state.randint(0, num_iterations / 2) == 0: dbm.Synchronize(False).OrDie() elif rnd_state.randint(0, 100) == 0: it = dbm.MakeIterator() if dbm.IsOrdered() and rnd_state.randint(0, 3) == 0: if rnd_state.randint(0, 3) == 0: it.Jump(key) else: it.Last() while rnd_state.randint(0, 10) == 0: status = Status() it.Get(status) if status != Status.NOT_FOUND_ERROR: status.OrDie() it.Previous() else: if rnd_state.randint(0, 3) == 0: it.Jump(key) else: it.First() while rnd_state.randint(0, 10) == 0: status = Status() it.Get(status) if status != Status.NOT_FOUND_ERROR: status.OrDie() it.Next() elif rnd_state.randint(0, 3) == 0: status = Status() dbm.Get(key, status) if status != Status.NOT_FOUND_ERROR: status.OrDie() elif rnd_state.randint(0, 3) == 0: status = dbm.Remove(key) if status != Status.NOT_FOUND_ERROR: status.OrDie() elif rnd_state.randint(0, 3) == 0: status = dbm.Set(key, value, False) if status != Status.DUPLICATION_ERROR: status.OrDie() else: dbm.Set(key, value).OrDie() seq = i + 1 if self.thid == 0 and seq % (num_iterations / 500) == 0: print(".", end="") if seq % (num_iterations / 10) == 0: print(" ({:08d})".format(seq)) sys.stdout.flush() print("Doing:") start_time = time.time() threads = [] for thid in range(0, num_threads): th = Task(thid) th.start() threads.append(th) for th in threads: th.join() dbm.Synchronize(False).OrDie() end_time = time.time() elapsed = end_time - start_time mem_usage = Utility.GetMemoryUsage() - start_mem_usage print("Done: num_records={:d} file_size={:d} time={:.3f} qps={:.0f} mem={:d}".format( dbm.Count(), dbm.GetFileSize() or -1, elapsed, num_iterations * num_threads / elapsed, mem_usage)) print("") dbm.Close().OrDie() return 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:])) # END OF FILE tkrzw-python-0.1.32/coro.py0000775000175000017500000000626614626642264014646 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Coroutine tests # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import asyncio import time import tkrzw NUM_ITERS = 100 NUM_COROUTINES = 1000 NUM_REPEATS = 100 async def set_sync(dbm, cor_id, iter_count): for i in range(0, NUM_REPEATS): key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i) dbm.Set(key, key).OrDie() async def set_async(adbm, cor_id, iter_count): futures = [] for i in range(0, NUM_REPEATS): key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i) futures.append(adbm.Set(key, key)) for future in futures: future.Get().OrDie() async def get_sync(dbm, cor_id, iter_count): for i in range(0, NUM_REPEATS): key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i) status = tkrzw.Status() dbm.Get(key, status) status.OrDie() async def get_async(adbm, cor_id, iter_count): futures = [] for i in range(0, NUM_REPEATS): key = str(cor_id * NUM_COROUTINES * NUM_REPEATS + iter_count * NUM_REPEATS + i) futures.append(adbm.Get(key)) for future in futures: status, value = future.Get() status.OrDie() async def main(): dbm = tkrzw.DBM() num_buckets = NUM_ITERS * NUM_COROUTINES * NUM_REPEATS / 2 dbm.Open("casket.tkh", True, concurrent=True, truncate=True, num_buckets=num_buckets, file="pos-para") adbm = tkrzw.AsyncDBM(dbm, 4) def make_set_sync(cor_id, iter_count): return set_sync(dbm, cor_id, iter_count) def make_set_async(cor_id, iter_count): return set_async(adbm, cor_id, iter_count) def make_get_sync(cor_id, iter_count): return get_sync(dbm, cor_id, iter_count) def make_get_async(cor_id, iter_count): return get_async(adbm, cor_id, iter_count) confs = [ {"label": "SET SYNC", "op": make_set_sync}, {"label": "SET ASYNC", "op": make_set_async}, {"label": "GET SYNC", "op": make_get_sync}, {"label": "GET ASYNC", "op": make_get_async}, ] for conf in confs: start_time = time.time() for iter_count in range(0, NUM_ITERS): coroutines = [] for cor_id in range(0, NUM_COROUTINES): coroutines.append(conf["op"](cor_id, iter_count)) for coroutine in coroutines: await coroutine end_time = time.time() elapsed = end_time - start_time print("{:10s}: {:8.0f} QPS".format( conf["label"], (NUM_ITERS * NUM_COROUTINES * NUM_REPEATS) / elapsed)) adbm.Destruct() dbm.Close() asyncio.run(main()) # END OF FILE tkrzw-python-0.1.32/CONTRIBUTING.md0000664000175000017500000000211114626642264015541 0ustar mikiomikio# How to Contribute We'd love to accept your patches and contributions to this project. There are just a few small guidelines you need to follow. ## Contributor License Agreement Contributions to this project must be accompanied by a Contributor License Agreement. You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project. Head over to to see your current agreements on file or to sign a new one. You generally only need to submit a CLA once, so if you've already submitted one (even if it was for a different project), you probably don't need to do it again. ## Code reviews All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more information on using pull requests. ## Community Guidelines This project follows [Google's Open Source Community Guidelines](https://opensource.google/conduct/). tkrzw-python-0.1.32/Makefile0000664000175000017500000001031014626642264014750 0ustar mikiomikio# Makefile for Tkrzw for Python PACKAGE = tkrzw-python VERSION = 0.1.32 PACKAGEDIR = $(PACKAGE)-$(VERSION) PACKAGETGZ = $(PACKAGE)-$(VERSION).tar.gz PYTHON = python3 RUNENV = LD_LIBRARY_PATH=.:/lib:/usr/lib:/usr/local/lib:$(HOME)/local/lib:$(HOME)/lib:$(LD_LIBRARY_PATH) all : $(PYTHON) setup.py build cp -f build/*/*.so . @printf '\n' @printf '#================================================================\n' @printf '# Ready to install.\n' @printf '#================================================================\n' clean : rm -rf casket casket* *~ *.tmp *.tkh *.tkt *.tks *.flat *.log *.so *.pyc build \ hoge moge tako ika uni install : $(PYTHON) setup.py install -chown -R `stat -c "%U:%G" Makefile` dist *.egg-info @printf '\n' @printf '#================================================================\n' @printf '# Thanks for using Tkrzw for Python.\n' @printf '#================================================================\n' uninstall : $(PYTHON) setup.py install --record files.tmp xargs rm -f < files.tmp dist : $(MAKE) distclean rm -Rf "../$(PACKAGEDIR)" "../$(PACKAGETGZ)" cd .. && cp -R tkrzw-python $(PACKAGEDIR) && \ tar --exclude=".*" -cvf - $(PACKAGEDIR) | gzip -c > $(PACKAGETGZ) rm -Rf "../$(PACKAGEDIR)" sync ; sync distclean : clean apidocclean rm -rf dist *.egg-info check : $(RUNENV) $(PYTHON) test.py $(RUNENV) $(PYTHON) perf.py --path casket.tkh --params "num_buckets=100000" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) perf.py --path casket.tkh --params "concurrent=true,num_buckets=100000" \ --iter 20000 --threads 5 --random $(RUNENV) $(PYTHON) perf.py --path casket.tkt --params "key_comparator=decimal" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) perf.py --path casket.tkt --params "concurrent=true,key_comparator=decimal" \ --iter 20000 --threads 5 --random $(RUNENV) $(PYTHON) perf.py --path casket.tks --params "step_unit=3" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) perf.py --path casket.tks --params "concurrent=true,step_unit=3" \ --iter 20000 --threads 5 --random $(RUNENV) $(PYTHON) perf.py --params "dbm=tiny,num_buckets=100000" \ --iter 20000 --threads 5 --random $(RUNENV) $(PYTHON) perf.py --params "dbm=baby,key_comparator=decimal" \ --iter 20000 --threads 5 --random $(RUNENV) $(PYTHON) perf.py --params "dbm=stdhash,num_buckets=100000" \ --iter 20000 --threads 5 --async_threads 2 --random $(RUNENV) $(PYTHON) perf.py --params "dbm=stdtree" \ --iter 20000 --threads 5 --async_threads 2 --random $(RUNENV) $(PYTHON) wicked.py --path casket.tkh --params "num_buckets=100000" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) wicked.py --path casket.tkt --params "key_comparator=decimal" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) wicked.py --path casket.tks --params "step_unit=3" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) wicked.py --params "dbm=tiny,num_buckets=100000" \ --iter 20000 --threads 5 $(RUNENV) $(PYTHON) wicked.py --params "dbm=baby,key_comparator=decimal" \ --iter 20000 --threads 5 @printf '\n' @printf '#================================================================\n' @printf '# Checking completed.\n' @printf '#================================================================\n' apidoc : $(MAKE) apidocclean mkdir -p tmp-doc cp tkrzw-doc.py tmp-doc/tkrzw.py cd tmp-doc ; sphinx-apidoc -F -H Tkrzw -A "Mikio Hirabayashi" -o out . cat tmp-doc/out/conf.py |\ sed -e 's/^# import /import /' -e 's/^# sys.path/sys.path/' \ -e 's/alabaster/haiku/' \ -e '/sphinx\.ext\.viewcode/d' \ -e '/^extensions = /a "sphinx.ext.autosummary",' > tmp-doc/out/conf.py.tmp echo >> tmp-doc/out/conf.py.tmp echo "autodoc_member_order = 'bysource'" >> tmp-doc/out/conf.py.tmp echo "html_title = 'Python binding of Tkrzw'" >> tmp-doc/out/conf.py.tmp echo "autodoc_default_options = {'members': True, 'special-members': True, 'exclude-members': '__dict__,__module__,__weakref__'}" >> tmp-doc/out/conf.py.tmp mv -f tmp-doc/out/conf.py.tmp tmp-doc/out/conf.py cp -f index.rst tmp-doc/out/index.rst cd tmp-doc/out ; $(MAKE) html mv tmp-doc/out/_build/html api-doc apidocclean : rm -rf api-doc tmp-doc .PHONY: all clean install uninstall dist distclean check apidoc apidocclean # END OF FILE tkrzw-python-0.1.32/example1.py0000775000175000017500000000330014626642264015402 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Example for basic usage of the hash database # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import tkrzw # Prepares the database. dbm = tkrzw.DBM() dbm.Open("casket.tkh", True, truncate=True, num_buckets=100) # Sets records. # If the operation fails, a runtime exception is raised. # Keys and values are implicitly converted into bytes. dbm["first"] = "hop" dbm["second"] = "step" dbm["third"] = "jump" # Retrieves record values. # If the operation fails, a runtime exception is raised. # Retrieved values are strings if keys are strings. print(dbm["first"]) print(dbm["second"]) print(dbm["third"]) try: print(dbm["fourth"]) except tkrzw.StatusException as e: print(repr(e)) # Checks and deletes a record. if "first" in dbm: del dbm["first"] # Traverses records. # Retrieved keys and values are always bytes so we decode them. for key, value in dbm: print(key.decode(), value.decode()) # Closes the database. dbm.Close() # END OF FILE tkrzw-python-0.1.32/COPYING0000664000175000017500000002613614626642264014360 0ustar mikiomikio Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. tkrzw-python-0.1.32/ChangeLog0000664000175000017500000000776614626642264015107 0ustar mikiomikio2024-05-30 Mikio Hirabayashi - Release: 0.1.32 - Add serializeInt and serializeFloat. - Support the floating-point number key comparator. - Compatibility to the C++ library: Tkrzw 1.0.30 2023-04-25 Mikio Hirabayashi - Release: 0.1.31 - The Index class is added. - Compatibility to the C++ library: Tkrzw 1.0.29 2023-03-11 Mikio Hirabayashi - Release: 0.1.30 - Options to enable encrypted databases are added. - Compatibility to the C++ library: Tkrzw 1.0.27 2023-02-22 Mikio Hirabayashi - Release: 0.1.29 - Process, ProcessMulti, and ProcessEach are added to DBM. 2021-11-13 Mikio Hirabayashi - Release: 0.1.28 - Build setting is modified for separate atomic libraries. - One database object can be opened/closed multiple times. - Add __contains__ method for "in" operator. 2021-11-02 Mikio Hirabayashi - Release: 0.1.27 - DBM::CompareExchangeAndGet is added. - Compatibility to the C++ library: Tkrzw 1.0.20 2021-10-30 Mikio Hirabayashi - Release: 0.1.26 - DBM::CompareExchange supports ANY_DATA. - Compatibility to the C++ library: Tkrzw 1.0.19 2021-10-12 Mikio Hirabayashi - Release: 0.1.25 - Add DBM::Rekey. - Add Iterator::Step. - Add DBM::PopFirst and DBM::PushLast. - Conversion from bytes to Unicode uses "replace" mode. - Compatibility to the C++ library: Tkrzw 1.0.18 2021-10-10 Mikio Hirabayashi - Release: 0.1.24 - lock_mem_buckets is ablished. - Compatibility to the C++ library: Tkrzw 1.0.17 - Add Status.CodeName. 2021-09-20 Mikio Hirabayashi - Release: 0.1.23 - sync_hard of the open options is supported. - Compatibility to the C++ library: Tkrzw 1.0.12 2021-09-19 Mikio Hirabayashi - Release: 0.1.22 - Add GetTimestamp method. - Compatibility to the C++ library: Tkrzw 1.0.11 2021-08-27 Mikio Hirabayashi - Release: 0.1.21 - Compatibility to the C++ library: Tkrzw 1.0.7 2021-08-21 Mikio Hirabayashi - Release: 0.1.20 - Compatibility to the C++ library: Tkrzw 1.0.2 2021-08-02 Mikio Hirabayashi - Release: 0.1.19 - Future#Get releases the resource. 2021-07-29 Mikio Hirabayashi - Release: 0.1.18 - Add Join method to Status. - Add AsyncDBM class. 2021-07-28 Mikio Hirabayashi - Release: 0.1.17 - Modify export methods. - Compatibility to the C++ library: Tkrzw 0.9.48 2021-07-25 Mikio Hirabayashi - Release: 0.1.16 - Add PAGE_SIZE constant. - Add File.GetPath method. - Compatibility to the C++ library: Tkrzw 0.9.47 2021-07-23 Mikio Hirabayashi - Release: 0.1.15 - Add IsWritable method. 2021-07-14 Mikio Hirabayashi - Release: 0.1.14 - Add AppendMulti method. - Compatibility to the C++ library: Tkrzw 0.9.43 2021-07-12 Mikio Hirabayashi - Release: 0.1.13 - Add exporting methods to the flat record file. - Compatibility to the C++ library: Tkrzw 0.9.42 2021-07-08 Mikio Hirabayashi - Release: 0.1.12 - Modify DBM::Search for the latest C++ API. - Compatibility to the C++ library: Tkrzw 0.9.40 2021-07-05 Mikio Hirabayashi - Release: 0.1.11 - The RestoreDatabase method is added. - TextFile is renamed to File. - Compatibility to the C++ library: Tkrzw 0.9.38 2021-06-02 Mikio Hirabayashi - Release: 0.1.10 - Add CompareExchangeMulti method. 2021-05-29 Mikio Hirabayashi - Release: 0.1.9 - Add RemoveMulti method. - CompareExchange supports expectation for missing records. - Compatibility to the C++ library: Tkrzw 0.9.21 2021-05-16 Mikio Hirabayashi - Release: 0.1.8 - Supports tunning parameters of Direct I/O. - Compatibility to the C++ library: Tkrzw 0.9.17 tkrzw-python-0.1.32/tkrzw-doc.py0000664000175000017500000020420114626642264015612 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # API document # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- class Utility: """ Library utilities. """ VERSION = "0.0.0" """The package version numbers.""" OS_NAME = "unknown" """The recognized OS name.""" PAGE_SIZE = 4096 """The size of a memory page on the OS.""" INT32MIN = -2 ** 31 """The minimum value of int32.""" INT32MAX = 2 ** 31 - 1 """The maximum value of int32.""" UINT32MAX = 2 ** 32 - 1 """The maximum value of uint32.""" INT64MIN = -2 ** 63 """The minimum value of int64.""" INT64MAX = 2 ** 63 - 1 """The maximum value of int64.""" UINT64MAX = 2 ** 64 - 1 """The maximum value of uint64.""" @classmethod def GetMemoryCapacity(cls): """ Gets the memory capacity of the platform. :return: The memory capacity of the platform in bytes, or -1 on failure. """ pass # native code @classmethod def GetMemoryUsage(cls): """ Gets the current memory usage of the process. :return: The current memory usage of the process in bytes, or -1 on failure. """ pass # native code @classmethod def PrimaryHash(cls, data, num_buckets=None): """ Primary hash function for the hash database. :param data: The data to calculate the hash value for. :param num_buckets: The number of buckets of the hash table. If it is omitted, UINT64MAX is set. :return: The hash value. """ pass # native code @classmethod def SecondaryHash(cls, data, num_shards=None): """ Secondary hash function for sharding. :param data: The data to calculate the hash value for. :param num_shards: The number of shards. If it is omitted, UINT64MAX is set. :return: The hash value. """ pass # native code @classmethod def EditDistanceLev(cls, a, b): """ Gets the Levenshtein edit distance of two Unicode strings. :param a: A Unicode string. :param b: The other Unicode string. :return: The Levenshtein edit distance of the two strings. """ pass # native code @classmethod def SerializeInt(cls, num): """ Serializes an integer into a big-endian binary sequence. :param num: an integer. :return: The result binary sequence. """ pass # native code @classmethod def DeserializeInt(cls, data): """ Deserializes a big-endian binary sequence into an integer. :param data: a binary sequence. :return: The result integer. """ pass # native code @classmethod def SerializeFloat(cls, num): """ Serializes a floating-point number into a big-endian binary sequence. :param num: a floating-point number. :return: The result binary sequence. """ pass # native code @classmethod def DeserializeFloat(cls, data): """ Deserializes a big-endian binary sequence into a floating-point number. :param data: a binary sequence. :return: The result floating-point number. """ pass # native code class Status: """ Status of operations. """ SUCCESS = 0 """Success.""" UNKNOWN_ERROR = 1 """Generic error whose cause is unknown.""" SYSTEM_ERROR = 2 """Generic error from underlying systems.""" NOT_IMPLEMENTED_ERROR = 3 """Error that the feature is not implemented.""" PRECONDITION_ERROR = 4 """Error that a precondition is not met.""" INVALID_ARGUMENT_ERROR = 5 """Error that a given argument is invalid.""" CANCELED_ERROR = 6 """Error that the operation is canceled.""" NOT_FOUND_ERROR = 7 """Error that a specific resource is not found.""" PERMISSION_ERROR = 8 """Error that the operation is not permitted.""" INFEASIBLE_ERROR = 9 """Error that the operation is infeasible.""" DUPLICATION_ERROR = 10 """Error that a specific resource is duplicated.""" BROKEN_DATA_ERROR = 11 """Error that internal data are broken.""" NETWORK_ERROR = 12 """Error caused by networking failure.""" APPLICATION_ERROR = 13 """Generic error caused by the application logic.""" def __init__(self, code=SUCCESS, message=""): """ Sets the code and the message. :param code: The status code. This can be omitted and then SUCCESS is set. :param message: An arbitrary status message. This can be omitted and the an empty string is set. """ pass # native code def __repr__(self): """ Returns a string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns a string representation of the content. :return: The string representation of the content. """ pass # native code def __eq__(self, rhs): """ Returns true if the given object is equivalent to this object. :return: True if the given object is equivalent to this object. This supports comparison between a status object and a status code number. """ pass # native code def Set(self, code=SUCCESS, message=""): """ Sets the code and the message. :param code: The status code. This can be omitted and then SUCCESS is set. :param message: An arbitrary status message. This can be omitted and the an empty string is set. """ pass # native code def Join(self, rht): """ Assigns the internal state from another status object only if the current state is success. :param rhs: The status object. """ pass # native code def GetCode(self): """ Gets the status code. :return: The status code. """ pass # native code def GetMessage(self): """ Gets the status message. :return: The status message. """ pass # native code def IsOK(self): """ Returns true if the status is success. :return: True if the status is success, or False on failure. """ pass # native code def OrDie(self): """ Raises an exception if the status is not success. :raise StatusException: An exception containing the status object. """ pass # native code @classmethod def CodeName(cls, code): """ Gets the string name of a status code. :param: code The status code. :return: The name of the status code. """ pass # native code class Future: """ Future containing a status object and extra data. Future objects are made by methods of AsyncDBM. Every future object should be destroyed by the "Destruct" method or the "Get" method to free resources. This class implements the awaitable protocol so an instance is usable with the "await" sentence. """ def __init__(self): """ The constructor cannot be called directly. Use methods of AsyncDBM. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns a string representation of the content. :return: The string representation of the content. """ pass # native code def __await__(self): """ Waits for the operation to be done and returns an iterator. :return: The iterator which stops immediately. """ pass # native code def Wait(self, timeout=-1): """ Waits for the operation to be done. :param timeout: The waiting time in seconds. If it is negative, no timeout is set. :return: True if the operation has done. False if timeout occurs. """ pass # native code def Get(self): """ Waits for the operation to be done and gets the result status. :return: The result status and extra data if any. The existence and the type of extra data depends on the operation which makes the future. For DBM#Get, a tuple of the status and the retrieved value is returned. For DBM#Set and DBM#Remove, the status object itself is returned. The internal resource is released by this method. "Wait" and "Get" cannot be called after calling this method. """ pass # native code class StatusException(RuntimeError): """ Exception to convey the status of operations. """ def __init__(self, status): """ Sets the status. :param status: The status object. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns A string representation of the content. :return: The string representation of the content. """ pass # native code def GetStatus(self): """ Gets the status object :return: The status object. """ pass # native code class DBM: """ Polymorphic database manager. All operations except for Open and Close are thread-safe; Multiple threads can access the same database concurrently. You can specify a data structure when you call the Open method. Every opened database must be closed explicitly by the Close method to avoid data corruption. This class implements the iterable protocol so an instance is usable with "for-in" loop. """ ANY_DATA = b"\x00[ANY]\x00" """The special bytes value for no-operation or any data.""" def __init__(self): """ Does nothing especially. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns A string representation of the content. :return: The string representation of the content. """ pass # native code def __len__(self): """ Gets the number of records, to enable the len operator. :return: The number of records on success, or 0 on failure. """ pass # native code def __getitem__(self, key): """ Gets the value of a record, to enable the [] operator. :param key: The key of the record. :return: The value of the matching record. An exception is raised for missing records. If the given key is a string, the returned value is also a string. Otherwise, the return value is bytes. :raise StatusException: An exception containing the status object. """ pass # native code def __contains__(self, key): """ Checks if a record exists or not, to enable the in operator. :param key: The key of the record. :return: True if the record exists, or False if not. No exception is raised for missing records. """ pass # native code def __setitem__(self, key, value): """ Sets a record of a key and a value, to enable the []= operator. :param key: The key of the record. :param value: The value of the record. :raise StatusException: An exception containing the status object. """ pass # native code def __delitem__(self, key): """ Removes a record of a key, to enable the del [] operator. :param key: The key of the record. :raise StatusException: An exception containing the status object. """ pass # native code def __iter__(self): """ Makes an iterator and initialize it, to comply to the iterator protocol. :return: The iterator for each record. """ pass # native code def Open(self, path, writable, **params): """ Opens a database file. :param path: A path of the file. :param writable: If true, the file is writable. If false, it is read-only. :param params: Optional keyword parameters. :return: The result status. The extension of the path indicates the type of the database. - .tkh : File hash database (HashDBM) - .tkt : File tree database (TreeDBM) - .tks : File skip database (SkipDBM) - .tkmt : On-memory hash database (TinyDBM) - .tkmb : On-memory tree database (BabyDBM) - .tkmc : On-memory cache database (CacheDBM) - .tksh : On-memory STL hash database (StdHashDBM) - .tkst : On-memory STL tree database (StdTreeDBM) The optional parameters can include an option for the concurrency tuning. By default, database operatins are done under the GIL (Global Interpreter Lock), which means that database operations are not done concurrently even if you use multiple threads. If the "concurrent" parameter is true, database operations are done outside the GIL, which means that database operations can be done concurrently if you use multiple threads. However, the downside is that swapping thread data is costly so the actual throughput is often worse in the concurrent mode than in the normal mode. Therefore, the concurrent mode should be used only if the database is huge and it can cause blocking of threads in multi-thread usage. The optional parameters can include options for the file opening operation. - truncate (bool): True to truncate the file. - no_create (bool): True to omit file creation. - no_wait (bool): True to fail if the file is locked by another process. - no_lock (bool): True to omit file locking. - sync_hard (bool): True to do physical synchronization when closing. The optional parameter "dbm" supercedes the decision of the database type by the extension. The value is the type name: "HashDBM", "TreeDBM", "SkipDBM", "TinyDBM", "BabyDBM", "CacheDBM", "StdHashDBM", "StdTreeDBM". The optional parameter "file" specifies the internal file implementation class. The default file class is "MemoryMapAtomicFile". The other supported classes are "StdFile", "MemoryMapAtomicFile", "PositionalParallelFile", and "PositionalAtomicFile". For HashDBM, these optional parameters are supported. - update_mode (string): How to update the database file: "UPDATE_IN_PLACE" for the in-palce or "UPDATE_APPENDING" for the appending mode. - record_crc_mode (string): How to add the CRC data to the record: "RECORD_CRC_NONE" to add no CRC to each record, "RECORD_CRC_8" to add CRC-8 to each record, "RECORD_CRC_16" to add CRC-16 to each record, or "RECORD_CRC_32" to add CRC-32 to each record. - record_comp_mode (string): How to compress the record data: "RECORD_COMP_NONE" to do no compression, "RECORD_COMP_ZLIB" to compress with ZLib, "RECORD_COMP_ZSTD" to compress with ZStd, "RECORD_COMP_LZ4" to compress with LZ4, "RECORD_COMP_LZMA" to compress with LZMA, "RECORD_COMP_RC4" to cipher with RC4, "RECORD_COMP_AES" to cipher with AES. - offset_width (int): The width to represent the offset of records. - align_pow (int): The power to align records. - num_buckets (int): The number of buckets for hashing. - restore_mode (string): How to restore the database file: "RESTORE_SYNC" to restore to the last synchronized state, "RESTORE_READ_ONLY" to make the database read-only, or "RESTORE_NOOP" to do nothing. By default, as many records as possible are restored. - fbp_capacity (int): The capacity of the free block pool. - min_read_size (int): The minimum reading size to read a record. - cache_buckets (bool): True to cache the hash buckets on memory. - cipher_key (string): The encryption key for cipher compressors. For TreeDBM, all optional parameters for HashDBM are available. In addition, these optional parameters are supported. - max_page_size (int): The maximum size of a page. - max_branches (int): The maximum number of branches each inner node can have. - max_cached_pages (int): The maximum number of cached pages. - page_update_mode (string): What to do when each page is updated: "PAGE_UPDATE_NONE" is to do no operation or "PAGE_UPDATE_WRITE" is to write immediately. - key_comparator (string): The comparator of record keys: "LexicalKeyComparator" for the lexical order, "LexicalCaseKeyComparator" for the lexical order ignoring case, "DecimalKeyComparator" for the order of decimal integer numeric expressions, "HexadecimalKeyComparator" for the order of hexadecimal integer numeric expressions, "RealNumberKeyComparator" for the order of decimal real number expressions, and "FloatBigEndianKeyComparator" for the order of binary float-number expressions. For SkipDBM, these optional parameters are supported. - offset_width (int): The width to represent the offset of records. - step_unit (int): The step unit of the skip list. - max_level (int): The maximum level of the skip list. - restore_mode (string): How to restore the database file: "RESTORE_SYNC" to restore to the last synchronized state or "RESTORE_NOOP" to do nothing make the database read-only. By default, as many records as possible are restored. - sort_mem_size (int): The memory size used for sorting to build the database in the at-random mode. - insert_in_order (bool): If true, records are assumed to be inserted in ascending order of the key. - max_cached_records (int): The maximum number of cached records. For TinyDBM, these optional parameters are supported. - num_buckets (int): The number of buckets for hashing. For BabyDBM, these optional parameters are supported. - key_comparator (string): The comparator of record keys. The same ones as TreeDBM. For CacheDBM, these optional parameters are supported. - cap_rec_num (int): The maximum number of records. - cap_mem_size (int): The total memory size to use. All databases support taking update logs into files. It is enabled by setting the prefix of update log files. - ulog_prefix (str): The prefix of the update log files. - ulog_max_file_size (num): The maximum file size of each update log file. By default, it is 1GiB. - ulog_server_id (num): The server ID attached to each log. By default, it is 0. - ulog_dbm_index (num): The DBM index attached to each log. By default, it is 0. For the file "PositionalParallelFile" and "PositionalAtomicFile", these optional parameters are supported. - block_size (int): The block size to which all blocks should be aligned. - access_options (str): Values separated by colon. "direct" for direct I/O. "sync" for synchrnizing I/O, "padding" for file size alignment by padding, "pagecache" for the mini page cache in the process. If the optional parameter "num_shards" is set, the database is sharded into multiple shard files. Each file has a suffix like "-00003-of-00015". If the value is 0, the number of shards is set by patterns of the existing files, or 1 if they doesn't exist. """ pass # native code def Close(self): """ Closes the database file. :return: The result status. """ pass # native code def Process(self, key, func, writable): """ Processes a record with an arbitrary function. :param key: The key of the record. :param func: The function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or None if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is None, the record is not modified. If the return value is False (not a false value but the False object), the record is removed. :param writable: True if the processor can edit the record. :return: The result status. This method is not available in the concurrent mode because the function cannot be invoked outside the GIL. """ pass # native code def Get(self, key, status=None): """ Gets the value of a record of a key. :param key: The key of the record. :param status: A status object to which the result status is assigned. It can be omitted. :return: The bytes value of the matching record or None on failure. """ pass # native code def GetStr(self, key, status=None): """ Gets the value of a record of a key, as a string. :param key: The key of the record. :param status: A status object to which the result status is assigned. It can be omitted. :return: The string value of the matching record or None on failure. """ pass # native code def GetMulti(self, *keys): """ Gets the values of multiple records of keys. :param keys: The keys of records to retrieve. :return: A map of retrieved records. Keys which don't match existing records are ignored. """ pass # native code def GetMultiStr(self, *keys): """ Gets the values of multiple records of keys, as strings. :param keys: The keys of records to retrieve. :return: A map of retrieved records. Keys which don't match existing records are ignored. """ pass # native code def Set(self, key, value, overwrite=True): """ Sets a record of a key and a value. :param key: The key of the record. :param value: The value of the record. :param overwrite: Whether to overwrite the existing value. It can be omitted and then false is set. :return: The result status. If overwriting is abandoned, DUPLICATION_ERROR is returned. """ pass # native code def SetMulti(self, overwrite=True, **records): """ Sets multiple records of the keyword arguments. :param overwrite: Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned. :param records: Records to store, specified as keyword parameters. :return: The result status. If there are records avoiding overwriting, DUPLICATION_ERROR is returned. """ pass # native code def SetAndGet(self, key, value, overwrite=True): """ Sets a record and get the old value. :param key: The key of the record. :param value: The value of the record. :param overwrite: Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned. :return: A pair of the result status and the old value. If the record has not existed when inserting the new record, None is assigned as the value. If not None, the type of the returned old value is the same as the parameter value. """ pass # native code def Remove(self, key): """ Removes a record of a key. :param key: The key of the record. :return: The result status. If there's no matching record, NOT_FOUND_ERROR is returned. """ pass # native code def RemoveMulti(self, keys): """ Removes records of keys. :param key: The keys of the records. :return: The result status. If there are missing records, NOT_FOUND_ERROR is returned. """ pass # native code def RemoveAndGet(self, key): """ Removes a record and get the value. :param key: The key of the record. :return: A pair of the result status and the record value. If the record does not exist, None is assigned as the value. If not None, the type of the returned value is the same as the parameter key. """ pass # native code def Append(self, key, value, delim=""): """ Appends data at the end of a record of a key. :param key: The key of the record. :param value: The value to append. :param delim: The delimiter to put after the existing record. :return: The result status. If there's no existing record, the value is set without the delimiter. """ pass # native code def AppendMulti(self, delim="", **records): """ Appends data to multiple records of the keyword arguments. :param delim: The delimiter to put after the existing record. :param records: Records to append, specified as keyword parameters. :return: The result status. If there's no existing record, the value is set without the delimiter. """ pass # native code def CompareExchange(self, key, expected, desired): """ Compares the value of a record and exchanges if the condition meets. :param key: The key of the record. :param expected: The expected value. If it is None, no existing record is expected. If it is ANY_DATA, an existing record with any value is expacted. :param desired: The desired value. If it is None, the record is to be removed. If it is ANY_DATA, no update is done. :return: The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned. """ pass # native code def CompareExchangeAndGet(self, key, expected, desired): """ Does compare-and-exchange and/or gets the old value of the record. :param key: The key of the record. :param expected: The expected value. If it is None, no existing record is expected. If it is ANY_DATA, an existing record with any value is expacted. :param desired: The desired value. If it is None, the record is to be removed. If it is ANY_DATA, no update is done. :return: A pair of the result status and the.old value of the record. If the condition doesn't meet, the state is INFEASIBLE_ERROR. If there's no existing record, the value is None. If not None, the type of the returned old value is the same as the expected or desired value. """ pass # native code def Increment(self, key, inc=1, init=0, status=None): """ Increments the numeric value of a record. :param key: The key of the record. :param inc: The incremental value. If it is Utility.INT64MIN, the current value is not changed and a new record is not created. :param init: The initial value. :param status: A status object to which the result status is assigned. It can be omitted. :return: The current value, or None on failure. The record value is stored as an 8-byte big-endian integer. Negative is also supported. """ pass # native code def CompareExchangeMulti(self, expected, desired): """ Compares the values of records and exchanges if the condition meets. :param expected: A sequence of pairs of the record keys and their expected values. If the value is None, no existing record is expected. If the value is ANY_DATA, an existing record with any value is expacted. :param desired: A sequence of pairs of the record keys and their desired values. If the value is None, the record is to be removed. :return: The result status. If the condition doesn't meet, INFEASIBLE_ERROR is returned. """ pass # native code def Rekey(old_key, new_key, overwrite=True, copying=False): """ Changes the key of a record. :param old_key: The old key of the record. :param new_key: The new key of the record. :param overwrite: Whether to overwrite the existing record of the new key. :param copying: Whether to retain the record of the old key. :return: The result status. If there's no matching record to the old key, NOT_FOUND_ERROR is returned. If the overwrite flag is false and there is an existing record of the new key, DUPLICATION ERROR is returned. This method is done atomically. The other threads observe that the record has either the old key or the new key. No intermediate states are observed. """ pass # native code def PopFirst(self, status=None): """ Gets the first record and removes it. :param status: A status object to which the result status is assigned. It can be omitted. :return: A tuple of the bytes key and the bytes value of the first record. On failure, None is returned. """ pass # native code def PopFirstStr(self, status=None): """ Gets the first record as strings and removes it. :param status: A status object to which the result status is assigned. It can be omitted. :return: A tuple of the string key and the string value of the first record. On failure, None is returned. """ pass # native code def PushLast(self, value, wtime=None): """ Adds a record with a key of the current timestamp. :param value: The value of the record. :param wtime: The current wall time used to generate the key. If it is None, the system clock is used. :return: The result status. The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds. """ pass # native code def ProcessEach(self, func, writable): """ Processes each and every record in the database with an arbitrary function. :param func: The function to process a record. The first parameter is the key of the record. The second parameter is the value of the existing record, or None if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is None, the record is not modified. If the return value is False (not a false value but the False object), the record is removed. :param writable: True if the processor can edit the record. :return: The result status. The given function is called repeatedly for each record. It is also called once before the iteration and once after the iteration with both the key and the value being None. This method is not available in the concurrent mode because the function cannot be invoked outside the GIL. """ pass # native code def Count(self): """ Gets the number of records. :return: The number of records on success, or None on failure. """ pass # native code def GetFileSize(self): """ Gets the current file size of the database. :return: The current file size of the database, or None on failure. """ pass # native code def GetFilePath(self): """ Gets the path of the database file. :return: The file path of the database, or None on failure. """ pass # native code def GetTimestamp(self): """ Gets the timestamp in seconds of the last modified time. :return: The timestamp of the last modified time, or None on failure. """ pass # native code def Clear(self): """ Removes all records. :return: The result status. """ pass # native code def Rebuild(self, **params): """ Rebuilds the entire database. :param params: Optional keyword parameters. :return: The result status. The optional parameters are the same as the Open method. Omitted tuning parameters are kept the same or implicitly optimized. In addition, HashDBM, TreeDBM, and SkipDBM supports the following parameters. - skip_broken_records (bool): If true, the operation continues even if there are broken records which can be skipped. - sync_hard (bool): If true, physical synchronization with the hardware is done before finishing the rebuilt file. """ pass # native code def ShouldBeRebuilt(self): """ Checks whether the database should be rebuilt. :return: True to be optimized or false with no necessity. """ pass # native code def Synchronize(self, hard, **params): """ Synchronizes the content of the database to the file system. :param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. :param params: Optional keyword parameters. :return: The result status. Only SkipDBM uses the optional parameters. The "merge" parameter specifies paths of databases to merge, separated by colon. The "reducer" parameter specifies the reducer to apply to records of the same key. "ReduceToFirst", "ReduceToSecond", "ReduceToLast", etc are supported. """ pass # native code def CopyFileData(self, dest_path, sync_hard=False): """ Copies the content of the database file to another file. :param dest_path: A path to the destination file. :param sync_hard: True to do physical synchronization with the hardware. :return: The result status. """ pass # native code def Export(self, dest_dbm): """ Exports all records to another database. :param dest_dbm: The destination database. :return: The result status. """ pass # native code def ExportToFlatRecords(self, dest_file): """ Exports all records of a database to a flat record file. :param dest_file: The file object to write records in. :return: The result status. A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration. """ pass # native code def ImportFromFlatRecords(self, src_file): """ Imports records to a database from a flat record file. :param src_file: The file object to read records from. :return: The result status. """ pass # native code def ExportKeysAsLines(self, dest_file): """ Exports the keys of all records as lines to a text file. :param dest_file: The file object to write keys in. :return: The result status. As the exported text file is smaller than the database file, scanning the text file by the search method is often faster than scanning the whole database. """ pass # native code def Inspect(self): """ Inspects the database. :return: A map of property names and their values. """ pass # native code def IsOpen(self): """ Checks whether the database is open. :return: True if the database is open, or false if not. """ pass # native code def IsWritable(self): """ Checks whether the database is writable. :return: True if the database is writable, or false if not. """ pass # native code def IsHealthy(self): """ Checks whether the database condition is healthy. :return: True if the database condition is healthy, or false if not. """ pass # native code def IsOrdered(self): """ Checks whether ordered operations are supported. :return: True if ordered operations are supported, or false if not. """ pass # native code def Search(self, mode, pattern, capacity=0): """ Searches the database and get keys which match a pattern. :param mode: The search mode. "contain" extracts keys containing the pattern. "begin" extracts keys beginning with the pattern. "end" extracts keys ending with the pattern. "regex" extracts keys partially matches the pattern of a regular expression. "edit" extracts keys whose edit distance to the UTF-8 pattern is the least. "editbin" extracts keys whose edit distance to the binary pattern is the least. "containcase", "containword", and "containcaseword" extract keys considering case and word boundary. Ordered databases support "upper" and "lower" which extract keys whose positions are upper/lower than the pattern. "upperinc" and "lowerinc" are their inclusive versions. :param pattern: The pattern for matching. :param capacity: The maximum records to obtain. 0 means unlimited. :return: A list of string keys matching the condition. """ pass # native code def MakeIterator(self): """ Makes an iterator for each record. :return: The iterator for each record. """ pass # native code @classmethod def RestoreDatabase(cls, old_file_path, new_file_path, class_name="", end_offset=-1, cipher_key=None): """ Restores a broken database as a new healthy database. :param old_file_path: The path of the broken database. :param new_file_path: The path of the new database to be created. :param class_name: The name of the database class. If it is None or empty, the class is guessed from the file extension. :param end_offset: The exclusive end offset of records to read. Negative means unlimited. 0 means the size when the database is synched or closed properly. Using a positive value is not meaningful if the number of shards is more than one. :param cipher_key: The encryption key for cipher compressors. If it is None, an empty key is used. :return: The result status. """ class Iterator: """ Iterator for each record. """ def __init__(self, dbm): """ Initializes the iterator. :param dbm: The database to scan. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns A string representation of the content. :return: The string representation of the content. """ pass # native code def __next__(self): """ Moves the iterator to the next record, to comply to the iterator protocol. :return: A tuple of The key and the value of the current record. """ pass # native code def First(self): """ Initializes the iterator to indicate the first record. :return: The result status. Even if there's no record, the operation doesn't fail. """ pass # native code def Last(self): """ Initializes the iterator to indicate the last record. :return: The result status. Even if there's no record, the operation doesn't fail. This method is suppoerted only by ordered databases. """ pass # native code def Jump(self, key): """ Initializes the iterator to indicate a specific record. :param key: The key of the record to look for. :return: The result status. Ordered databases can support "lower bound" jump; If there's no record with the same key, the iterator refers to the first record whose key is greater than the given key. The operation fails with unordered databases if there's no record with the same key. """ pass # native code def JumpLower(self, key, inclusive=False): """ Initializes the iterator to indicate the last record whose key is lower than a given key. :param key: The key to compare with. :param inclusive: If true, the considtion is inclusive: equal to or lower than the key. :return: The result status. Even if there's no matching record, the operation doesn't fail. This method is suppoerted only by ordered databases. """ pass # native code def JumpUpper(self, key, inclusive=False): """ Initializes the iterator to indicate the first record whose key is upper than a given key. :param key: The key to compare with. :param inclusive: If true, the considtion is inclusive: equal to or upper than the key. :return: The result status. Even if there's no matching record, the operation doesn't fail. This method is suppoerted only by ordered databases. """ pass # native code def Next(self): """ Moves the iterator to the next record. :return: The result status. If the current record is missing, the operation fails. Even if there's no next record, the operation doesn't fail. """ pass # native code def Previous(self): """ Moves the iterator to the previous record. :return: The result status. If the current record is missing, the operation fails. Even if there's no previous record, the operation doesn't fail. This method is suppoerted only by ordered databases. """ pass # native code def Get(self, status=None): """ Gets the key and the value of the current record of the iterator. :param status: A status object to which the result status is assigned. It can be omitted. :return: A tuple of the bytes key and the bytes value of the current record. On failure, None is returned. """ pass # native code def GetStr(self, status=None): """ Gets the key and the value of the current record of the iterator, as strings. :param status: A status object to which the result status is assigned. It can be omitted. :return: A tuple of the string key and the string value of the current record. On failure, None is returned. """ pass # native code def GetKey(self, status=None): """ Gets the key of the current record. :param status: A status object to which the result status is assigned. It can be omitted. :return: The bytes key of the current record or None on failure. """ pass # native code def GetKeyStr(self, status=None): """ Gets the key of the current record, as a string. :param status: A status object to which the result status is assigned. It can be omitted. :return: The string key of the current record or None on failure. """ pass # native code def GetValue(self, status=None): """ Gets the value of the current record. :param status: A status object to which the result status is assigned. It can be omitted. :return: The bytes value of the current record or None on failure. """ pass # native code def GetValueStr(self, status=None): """ Gets the value of the current record, as a string. :param status: A status object to which the result status is assigned. It can be omitted. :return: The string value of the current record or None on failure. """ pass # native code def Set(self, value): """ Sets the value of the current record. :param value: The value of the record. :return: The result status. """ pass # native code def Remove(self): """ Removes the current record. :return: The result status. """ pass # native code def Step(self, status=None): """ Gets the current record and moves the iterator to the next record. :param status: A status object to which the result status is assigned. It can be omitted. :return: A tuple of the bytes key and the bytes value of the current record. On failure, None is returned. """ pass # native code def StepStr(self, status=None): """ Gets the current record and moves the iterator to the next record, as strings. :param status: A status object to which the result status is assigned. It can be omitted. :return: A tuple of the string key and the string value of the current record. On failure, None is returned. """ pass # native code class AsyncDBM: """ Asynchronous database manager adapter. This class is a wrapper of DBM for asynchronous operations. A task queue with a thread pool is used inside. Every method except for the constructor and the destructor is run by a thread in the thread pool and the result is set in the future oject of the return value. The caller can ignore the future object if it is not necessary. The Destruct method waits for all tasks to be done. Therefore, the destructor should be called before the database is closed. """ def __init__(self, dbm, num_worker_threads): """ Sets up the task queue. :param dbm: A database object which has been opened. :param num_worker_threads: The number of threads in the internal thread pool. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns a string representation of the content. :return: The string representation of the content. """ pass # native code def Destruct(): """ Destructs the asynchronous database adapter. This method waits for all tasks to be done. """ def Get(self, key): """ Gets the value of a record of a key. :param key: The key of the record. :return: The future for the result status and the bytes value of the matching record. """ pass # native code def GetStr(self, key): """ Gets the value of a record of a key, as a string. :param key: The key of the record. :return: The future for the result status and the string value of the matching record. """ pass # native code def GetMulti(self, *keys): """ Gets the values of multiple records of keys. :param keys: The keys of records to retrieve. :return: The future for the result status and a map of retrieved records. Keys which don't match existing records are ignored. """ pass # native code def GetMultiStr(self, *keys): """ Gets the values of multiple records of keys, as strings. :param keys: The keys of records to retrieve. :return: The future for the result status and a map of retrieved records. Keys which don't match existing records are ignored. """ pass # native code def Set(self, key, value, overwrite=True): """ Sets a record of a key and a value. :param key: The key of the record. :param value: The value of the record. :param overwrite: Whether to overwrite the existing value. It can be omitted and then false is set. :return: The future for the result status. If overwriting is abandoned, DUPLICATION_ERROR is set. """ pass # native code def SetMulti(self, overwrite=True, **records): """ Sets multiple records of the keyword arguments. :param overwrite: Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is overwritten by the new value. If false, the operation is given up and an error status is returned. :param records: Records to store, specified as keyword parameters. :return: The future for the result status. If overwriting is abandoned, DUPLICATION_ERROR is set. """ pass # native code def Append(self, key, value, delim=""): """ Appends data at the end of a record of a key. :param key: The key of the record. :param value: The value to append. :param delim: The delimiter to put after the existing record. :return: The future for the result status. If there's no existing record, the value is set without the delimiter. """ pass # native code def AppendMulti(self, delim="", **records): """ Appends data to multiple records of the keyword arguments. :param delim: The delimiter to put after the existing record. :param records: Records to append, specified as keyword parameters. :return: The future for the result status. If there's no existing record, the value is set without the delimiter. """ pass # native code def CompareExchange(self, key, expected, desired): """ Compares the value of a record and exchanges if the condition meets. :param key: The key of the record. :param expected: The expected value. If it is None, no existing record is expected. If it is DBM.ANY_DATA, an existing record with any value is expacted. :param desired: The desired value. If it is None, the record is to be removed. If it is None, the record is to be removed. If it is DBM.ANY_DATA, no update is done. :return: The future for the result status. If the condition doesn't meet, INFEASIBLE_ERROR is set. """ pass # native code def Increment(self, key, inc=1, init=0): """ Increments the numeric value of a record. :param key: The key of the record. :param inc: The incremental value. If it is Utility.INT64MIN, the current value is not changed and a new record is not created. :param init: The initial value. :return: The future for the result status and the current value. The record value is stored as an 8-byte big-endian integer. Negative is also supported. """ pass # native code def ProcessMulti(self, key_func_pairs, writable): """ Processes multiple records with arbitrary functions. :param key_func_pairs: A list of pairs of keys and their functions. The first parameter of the function is the key of the record. The second parameter is the value of the existing record, or None if it the record doesn't exist. The return value is a string or bytes to update the record value. If the return value is None, the record is not modified. If the return value is False (not a false value but the False object), the record is removed. :param writable: True if the processors can edit the record. :return: The result status. This method is not available in the concurrent mode because the function cannot be invoked outside the GIL. """ pass # native code def CompareExchangeMulti(self, expected, desired): """ Compares the values of records and exchanges if the condition meets. :param expected: A sequence of pairs of the record keys and their expected values. If the value is None, no existing record is expected. If the value is DBM.ANY_DATA, an existing record with any value is expacted. :param desired: A sequence of pairs of the record keys and their desired values. If the value is None, the record is to be removed. :return: The future for the result status. If the condition doesn't meet, INFEASIBLE_ERROR is set. """ pass # native code def Rekey(old_key, new_key, overwrite=True, copying=False): """ Changes the key of a record. :param old_key: The old key of the record. :param new_key: The new key of the record. :param overwrite: Whether to overwrite the existing record of the new key. :param copying: Whether to retain the record of the old key. :return: The future for the result status. If there's no matching record to the old key, NOT_FOUND_ERROR is set. If the overwrite flag is false and there is an existing record of the new key, DUPLICATION ERROR is set. This method is done atomically. The other threads observe that the record has either the old key or the new key. No intermediate states are observed. """ pass # native code def PopFirst(self): """ Gets the first record and removes it. :return: The future for a tuple of the result status, the bytes key, and the bytes value of the first record. """ pass # native code def PopFirstStr(self): """ Gets the first record as strings and removes it. :return: The future for a tuple of the result status, the string key, and the string value of the first record. """ pass # native code def PushLast(self, value, wtime=None): """ Adds a record with a key of the current timestamp. :param value: The value of the record. :param wtime: The current wall time used to generate the key. If it is None, the system clock is used. :return: The future for the result status. The key is generated as an 8-bite big-endian binary string of the timestamp. If there is an existing record matching the generated key, the key is regenerated and the attempt is repeated until it succeeds. """ pass # native code def Clear(self): """ Removes all records. :return: The future for the result status. """ pass # native code def Rebuild(self, **params): """ Rebuilds the entire database. :param params: Optional keyword parameters. :return: The future for the result status. The parameters work in the same way as with DBM::Rebuild. """ pass # native code def Synchronize(self, hard, **params): """ Synchronizes the content of the database to the file system. :param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. :param params: Optional keyword parameters. :return: The future for the result status. The parameters work in the same way as with DBM::Synchronize. """ pass # native code def CopyFileData(self, dest_path, sync_hard=False): """ Copies the content of the database file to another file. :param dest_path: A path to the destination file. :param sync_hard: True to do physical synchronization with the hardware. :return: The future for the result status. """ pass # native code def Export(self, dest_dbm): """ Exports all records to another database. :param dest_dbm: The destination database. The lefetime of the database object must last until the task finishes. :return: The future for the result status. """ pass # native code def ExportToFlatRecords(self, dest_file): """ Exports all records of a database to a flat record file. :param dest_file: The file object to write records in. The lefetime of the file object must last until the task finishes. :return: The future for the result status. A flat record file contains a sequence of binary records without any high level structure so it is useful as a intermediate file for data migration. """ pass # native code def ImportFromFlatRecords(self, src_file): """ Imports records to a database from a flat record file. :param src_file: The file object to read records from. The lefetime of the file object must last until the task finishes. :return: The future for the result status. """ pass # native code def Search(self, mode, pattern, capacity=0): """ Searches the database and get keys which match a pattern. :param mode: The search mode. "contain" extracts keys containing the pattern. "begin" extracts keys beginning with the pattern. "end" extracts keys ending with the pattern. "regex" extracts keys partially matches the pattern of a regular expression. "edit" extracts keys whose edit distance to the UTF-8 pattern is the least. "editbin" extracts keys whose edit distance to the binary pattern is the least. :param pattern: The pattern for matching. :param capacity: The maximum records to obtain. 0 means unlimited. :return: The future for the result status and a list of keys matching the condition. """ pass # native code class File: """ Generic file implementation. All operations except for Open and Close are thread-safe; Multiple threads can access the same file concurrently. You can specify a concrete class when you call the Open method. Every opened file must be closed explicitly by the Close method to avoid data corruption. """ def __init__(self): """ Initializes the file object. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns A string representation of the content. :return: The string representation of the content. """ pass # native code def Open(self, path, writable, **params): """ Opens a file. :param path: A path of the file. :param writable: If true, the file is writable. If false, it is read-only. :param params: Optional keyword parameters. :return: The result status. The optional parameters can include an option for the concurrency tuning. By default, database operatins are done under the GIL (Global Interpreter Lock), which means that database operations are not done concurrently even if you use multiple threads. If the "concurrent" parameter is true, database operations are done outside the GIL, which means that database operations can be done concurrently if you use multiple threads. However, the downside is that swapping thread data is costly so the actual throughput is often worse in the concurrent mode than in the normal mode. Therefore, the concurrent mode should be used only if the database is huge and it can cause blocking of threads in multi-thread usage. The optional parameters can include options for the file opening operation. - truncate (bool): True to truncate the file. - no_create (bool): True to omit file creation. - no_wait (bool): True to fail if the file is locked by another process. - no_lock (bool): True to omit file locking. - sync_hard (bool): True to do physical synchronization when closing. The optional parameter "file" specifies the internal file implementation class. The default file class is "MemoryMapAtomicFile". The other supported classes are "StdFile", "MemoryMapAtomicFile", "PositionalParallelFile", and "PositionalAtomicFile". For the file "PositionalParallelFile" and "PositionalAtomicFile", these optional parameters are supported. - block_size (int): The block size to which all blocks should be aligned. - access_options (str): Values separated by colon. "direct" for direct I/O. "sync" for synchrnizing I/O, "padding" for file size alignment by padding, "pagecache" for the mini page cache in the process. """ pass # native code def Close(self): """ Closes the file. :return: The result status. """ pass # native code def Read(self, off, size, status=None): """ Reads data. :param off: The offset of a source region. :param size: The size to be read. :param status: A status object to which the result status is assigned. It can be omitted. :return: The bytes value of the read data or None on failure. """ pass # native code def ReadStr(self, off, size, status=None): """ Reads data as a string. :param off: The offset of a source region. :param size: The size to be read. :param status: A status object to which the result status is assigned. It can be omitted. :return: The string value of the read data or None on failure. """ pass # native code def Write(self, off, data): """ Writes data. :param off: The offset of the destination region. :param data: The data to write. :return: The result status. """ pass # native code def Append(self, data, status=None): """ Appends data at the end of the file. :param data: The data to write. :param status: A status object to which the result status is assigned. It can be omitted. :return: The offset at which the data has been put, or None on failure. """ pass # native code def Truncate(self, size): """ Truncates the file. :param size: The new size of the file. :return: The result status. If the file is shrunk, data after the new file end is discarded. If the file is expanded, null codes are filled after the old file end. """ pass # native code def Synchronize(self, hard, off=0, size=0): """ Synchronizes the content of the file to the file system. :param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. :param off: The offset of the region to be synchronized. :param size: The size of the region to be synchronized. If it is zero, the length to the end of file is specified. :return: The result status. The pysical file size can be larger than the logical size in order to improve performance by reducing frequency of allocation. Thus, you should call this function before accessing the file with external tools. """ pass # native code def GetSize(self): """ Gets the size of the file. :return: The size of the file or None on failure. """ pass # native code def GetPath(self): """ Gets the path of the file. :return: The path of the file or None on failure. """ pass # native code def Search(self, mode, pattern, capacity=0): """ Searches the file and get lines which match a pattern. :param mode: The search mode. "contain" extracts lines containing the pattern. "begin" extracts lines beginning with the pattern. "end" extracts lines ending with the pattern. "regex" extracts lines partially matches the pattern of a regular expression. "edit" extracts lines whose edit distance to the UTF-8 pattern is the least. "editbin" extracts lines whose edit distance to the binary pattern is the least. :param pattern: The pattern for matching. :param capacity: The maximum records to obtain. 0 means unlimited. :return: A list of lines matching the condition. """ pass # native code class Index: """ Secondary index interface. All operations except for Open and Close are thread-safe; Multiple threads can access the same index concurrently. You can specify a data structure when you call the Open method. Every opened index must be closed explicitly by the Close method to avoid data corruption. This class implements the iterable protocol so an instance is usable with "for-in" loop. """ def __repr__(self): """ Returns a string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns a string representation of the content. :return: The string representation of the content. """ pass # native code def __len__(self): """ Gets the number of records, to enable the len operator. :return: The number of records on success, or 0 on failure. """ pass # native code def __contains__(self, record): """ Checks if a record exists or not, to enable the in operator. :param record: A tuple of the key and the value to check. :return: True if the record exists, or False if not. No exception is raised for missing records. """ pass # native code def __iter__(self): """ Makes an iterator and initialize it, to comply to the iterator protocol. :return: The iterator for each record. """ pass # native code def Open(self, path, writable, **params): """ Opens an index file. :param path: A path of the file. :param writable: If true, the file is writable. If false, it is read-only. :param params: Optional keyword parameters. :return: The result status. If the path is empty, BabyDBM is used internally, which is equivalent to using the MemIndex class. If the path ends with ".tkt", TreeDBM is used internally, which is equivalent to using the FileIndex class. If the key comparator of the tuning parameter is not set, PairLexicalKeyComparator is set implicitly. Other compatible key comparators are PairLexicalCaseKeyComparator, PairDecimalKeyComparator, PairHexadecimalKeyComparator, PairRealNumberKeyComparator, and PairFloatBigEndianKeyComparator. Other options can be specified as with DBM::Open. """ pass # native code def Close(self): """ Closes the index file. :return: The result status. """ pass # native code def GetValues(self, key, max=0): """ Gets all values of records of a key. :param key: The key to look for. :param max: The maximum number of values to get. 0 means unlimited. :return: A list of all values of the key. An empty list is returned on failure. """ pass # native code def GetValuesStr(self, key, max=0): """ Gets all values of records of a key, as strings. :param key: The key to look for. :param max: The maximum number of values to get. 0 means unlimited. :return: A list of all values of the key. An empty list is returned on failure. """ pass # native code def Add(self, key, value): """ Adds a record. :param key: The key of the record. This can be an arbitrary expression to search the index. :param value: The value of the record. This should be a primary value of another database. :return: The result status. """ pass # native code def Remove(self, key, value): """ Removes a record. :param key: The key of the record. :param value: The value of the record. :return: The result status. """ pass # native code def Count(self): """ Gets the number of records. :return: The number of records, or 0 on failure. """ pass # native code def GetFilePath(self): """ Gets the path of the index file. :return: The file path of the index, or an empty string on failure. """ pass # native code def Clear(self): """ Removes all records. :return: The result status. """ pass # native code def Rebuild(self): """ Rebuilds the entire index. :return: The result status. """ pass # native code def Synchronize(self, hard): """ Synchronizes the content of the index to the file system. :param hard: True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. :return: The result status. """ pass # native code def IsOpen(self): """ Checks whether the index is open. :return: True if the index is open, or false if not. """ pass # native code def IsWritable(self): """ Checks whether the index is writable. :return: True if the index is writable, or false if not. """ def MakeIterator(self): """ Makes an iterator for each record. :return: The iterator for each record. """ pass # native code class IndexIterator: """ Iterator for each record of the secondary index. """ def __init__(self, index): """ Initializes the iterator. :param index: The index to scan. """ pass # native code def __repr__(self): """ Returns A string representation of the object. :return: The string representation of the object. """ pass # native code def __str__(self): """ Returns A string representation of the content. :return: The string representation of the content. """ pass # native code def __next__(self): """ Moves the iterator to the next record, to comply to the iterator protocol. :return: A tuple of The key and the value of the current record. """ pass # native code def First(self): """ Initializes the iterator to indicate the first record. """ pass # native code def Last(self): """ Initializes the iterator to indicate the last record. """ pass # native code def Jump(self, key, value=""): """ Initializes the iterator to indicate a specific range. :param key: The key of the lower bound. :param value: The value of the lower bound. """ pass # native code def Next(self): """ Moves the iterator to the next record. """ pass # native code def Previous(self): """ Moves the iterator to the previous record. """ pass # native code def Get(self): """ Gets the key and the value of the current record of the iterator. :return: A tuple of the bytes key and the bytes value of the current record. On failure, None is returned. """ pass # native code def GetStr(self): """ Gets the key and the value of the current record of the iterator, as strings. :return: A tuple of the string key and the string value of the current record. On failure, None is returned. """ pass # native code # END OF FILE tkrzw-python-0.1.32/tkrzw.cc0000664000175000017500000053027114626642264015015 0ustar mikiomikio/************************************************************************************************* * Python binding of Tkrzw * * Copyright 2020 Google LLC * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file * except in compliance with the License. You may obtain a copy of the License at * https://www.apache.org/licenses/LICENSE-2.0 * Unless required by applicable law or agreed to in writing, software distributed under the * License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, * either express or implied. See the License for the specific language governing permissions * and limitations under the License. *************************************************************************************************/ #include #include #include #include #include #include #include #include "tkrzw_cmd_util.h" #include "tkrzw_dbm.h" #include "tkrzw_dbm_common_impl.h" #include "tkrzw_dbm_poly.h" #include "tkrzw_dbm_shard.h" #include "tkrzw_file.h" #include "tkrzw_file_mmap.h" #include "tkrzw_file_poly.h" #include "tkrzw_file_util.h" #include "tkrzw_key_comparators.h" #include "tkrzw_lib_common.h" #include "tkrzw_str_util.h" extern "C" { #undef _POSIX_C_SOURCE #undef _XOPEN_SOURCE #include "Python.h" #include "structmember.h" // Global variables. PyObject* mod_tkrzw; PyObject* cls_utility; PyObject* cls_status; PyObject* cls_expt; PyObject* cls_future; PyObject* cls_dbm; PyObject* cls_iter; PyObject* cls_asyncdbm; PyObject* cls_file; PyObject* cls_index; PyObject* cls_indexiter; PyObject* obj_dbm_any_data; // Python object of Utility. struct PyUtility { PyObject_HEAD }; // Python object of Status. struct PyTkStatus { PyObject_HEAD tkrzw::Status* status; }; // Python object of Future. struct PyFuture { PyObject_HEAD tkrzw::StatusFuture* future; bool concurrent; bool is_str; }; // Python ofject of StatusException. struct PyException { PyException_HEAD PyObject* pystatus; }; // Python object of DBM. struct PyDBM { PyObject_HEAD tkrzw::ParamDBM* dbm; bool concurrent; }; // Python object of Iterator. struct PyIterator { PyObject_HEAD tkrzw::DBM::Iterator* iter; bool concurrent; }; // Python object of AsyncDBM. struct PyAsyncDBM { PyObject_HEAD tkrzw::AsyncDBM* async; bool concurrent; }; // Python object of File. struct PyFile { PyObject_HEAD tkrzw::PolyFile* file; bool concurrent; }; // Python object of Index. struct PyIndex { PyObject_HEAD tkrzw::PolyIndex* index; bool concurrent; }; // Python object of IndexIterator. struct PyIndexIterator { PyObject_HEAD tkrzw::PolyIndex::Iterator* iter; bool concurrent; }; // Creates a new string of Python. static PyObject* CreatePyString(std::string_view str) { return PyUnicode_DecodeUTF8(str.data(), str.size(), "replace"); } // Creatse a new byte array of Python. static PyObject* CreatePyBytes(std::string_view str) { return PyBytes_FromStringAndSize(str.data(), str.size()); } // Creates a status object of Python. static PyObject* CreatePyTkStatus(const tkrzw::Status& status) { PyTypeObject* pytype = (PyTypeObject*)cls_status; PyTkStatus* obj = (PyTkStatus*)pytype->tp_alloc(pytype, 0); if (!obj) return nullptr; obj->status = new tkrzw::Status(status); return (PyObject*)obj; } // Creates a status object of Python, in moving context. static PyObject* CreatePyTkStatusMove(tkrzw::Status&& status) { PyTypeObject* pytype = (PyTypeObject*)cls_status; PyTkStatus* obj = (PyTkStatus*)pytype->tp_alloc(pytype, 0); if (!obj) return nullptr; obj->status = new tkrzw::Status(std::move(status)); return (PyObject*)obj; } // Creates a status future object of Python, in moving context. static PyObject* CreatePyFutureMove( tkrzw::StatusFuture&& future, bool concurrent, bool is_str = false) { PyTypeObject* pytype = (PyTypeObject*)cls_future; PyFuture* obj = (PyFuture*)pytype->tp_alloc(pytype, 0); if (!obj) return nullptr; obj->future = new tkrzw::StatusFuture(std::move(future)); obj->concurrent = concurrent; obj->is_str = is_str; return (PyObject*)obj; } // Throws an invalid argument error. static void ThrowInvalidArguments(std::string_view message) { PyErr_SetString(PyExc_TypeError, tkrzw::StrCat("invalid arguments: ", message).c_str()); } // Throws a status error. static void ThrowStatusException(const tkrzw::Status& status) { PyObject* pystatus = CreatePyTkStatus(status); PyErr_SetObject(cls_expt, pystatus); Py_DECREF(pystatus); } // Locking device to call a nagive function. class NativeLock final { public: NativeLock(bool concurrent) : thstate_(nullptr) { if (concurrent) { thstate_ = PyEval_SaveThread(); } } ~NativeLock() { if (thstate_) { PyEval_RestoreThread(thstate_); } } void Release() { if (thstate_) { PyEval_RestoreThread(thstate_); } thstate_ = nullptr; } private: PyThreadState* thstate_; }; // Wrapper to treat a Python string as a C++ string_view. class SoftString final { public: explicit SoftString(PyObject* pyobj) : pyobj_(pyobj), pystr_(nullptr), pybytes_(nullptr), ptr_(nullptr), size_(0) { Py_INCREF(pyobj_); if (PyUnicode_Check(pyobj_)) { pybytes_ = PyUnicode_AsUTF8String(pyobj_); if (pybytes_) { ptr_ = PyBytes_AS_STRING(pybytes_); size_ = PyBytes_GET_SIZE(pybytes_); } else { PyErr_Clear(); ptr_ = ""; size_ = 0; } } else if (PyBytes_Check(pyobj_)) { ptr_ = PyBytes_AS_STRING(pyobj_); size_ = PyBytes_GET_SIZE(pyobj_); } else if (PyByteArray_Check(pyobj_)) { ptr_ = PyByteArray_AS_STRING(pyobj_); size_ = PyByteArray_GET_SIZE(pyobj_); } else if (pyobj_ == Py_None) { ptr_ = ""; size_ = 0; } else { pystr_ = PyObject_Str(pyobj_); if (pystr_) { pybytes_ = PyUnicode_AsUTF8String(pystr_); if (pybytes_) { ptr_ = PyBytes_AS_STRING(pybytes_); size_ = PyBytes_GET_SIZE(pybytes_); } else { PyErr_Clear(); ptr_ = ""; size_ = 0; } } else { ptr_ = "(unknown)"; size_ = std::strlen(ptr_); } } } ~SoftString() { if (pybytes_) Py_DECREF(pybytes_); if (pystr_) Py_DECREF(pystr_); Py_DECREF(pyobj_); } std::string_view Get() const { return std::string_view(ptr_, size_); } private: PyObject* pyobj_; PyObject* pystr_; PyObject* pybytes_; const char* ptr_; size_t size_; }; // Converts a numeric parameter to an integer. static int64_t PyObjToInt(PyObject* pyobj) { if (PyLong_Check(pyobj)) { return PyLong_AsLong(pyobj); } else if (PyFloat_Check(pyobj)) { return (int64_t)PyFloat_AsDouble(pyobj); } else if (PyUnicode_Check(pyobj) || PyBytes_Check(pyobj)) { SoftString str(pyobj); return tkrzw::StrToInt(str.Get()); } else if (pyobj != Py_None) { int64_t num = 0; PyObject* pylong = PyNumber_Long(pyobj); if (pylong) { num = PyLong_AsLong(pylong); Py_DECREF(pylong); } return num; } return 0; } // Converts a numeric parameter to a double. static double PyObjToDouble(PyObject* pyobj) { if (PyLong_Check(pyobj)) { return PyLong_AsLong(pyobj); } else if (PyFloat_Check(pyobj)) { return PyFloat_AsDouble(pyobj); } else if (PyUnicode_Check(pyobj) || PyBytes_Check(pyobj)) { SoftString str(pyobj); return tkrzw::StrToDouble(str.Get()); } else if (pyobj != Py_None) { double num = 0; PyObject* pyfloat = PyNumber_Float(pyobj); if (pyfloat) { num = PyFloat_AsDouble(pyfloat); Py_DECREF(pyfloat); } return num; } return 0; } // Converts a Python Unicode object into a UCS-4 vector. std::vector PyUnicodeToUCS4(PyObject* pyuni) { const int32_t kind = PyUnicode_KIND(pyuni); void* data = PyUnicode_DATA(pyuni); const int32_t len = PyUnicode_GET_LENGTH(pyuni); std::vector ucs; ucs.reserve(len); for (int32_t i = 0; i < len; i++) { ucs.emplace_back(PyUnicode_READ(kind, data, i)); } return ucs; } // Sets a constant of long integer. static bool SetConstLong(PyObject* pyobj, const char* name, int64_t value) { PyObject* pyname = PyUnicode_FromString(name); PyObject* pyvalue = PyLong_FromLongLong(value); return PyObject_GenericSetAttr(pyobj, pyname, pyvalue) == 0; } // Sets a constant of unsigned long integer. static bool SetConstUnsignedLong(PyObject* pyobj, const char* name, uint64_t value) { PyObject* pyname = PyUnicode_FromString(name); PyObject* pyvalue = PyLong_FromUnsignedLongLong(value); return PyObject_GenericSetAttr(pyobj, pyname, pyvalue) == 0; } // Sets a constant of string. static bool SetConstStr(PyObject* pyobj, const char* name, const char* value) { PyObject* pyname = PyUnicode_FromString(name); PyObject* pyvalue = PyUnicode_FromString(value); return PyObject_GenericSetAttr(pyobj, pyname, pyvalue) == 0; } // Maps keyword arguments into C++ map. static std::map MapKeywords(PyObject* pykwds) { std::map map; PyObject* pykwlist = PyMapping_Items(pykwds); const int32_t kwnum = PyList_GET_SIZE(pykwlist); for (int32_t i = 0; i < kwnum; i++) { PyObject* pyrec = PyList_GET_ITEM(pykwlist, i); if (PyTuple_GET_SIZE(pyrec) == 2) { PyObject* pykey = PyTuple_GET_ITEM(pyrec, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyrec, 1); SoftString key(pykey); SoftString value(pyvalue); map.emplace(std::string(key.Get()), std::string(value.Get())); } } Py_DECREF(pykwlist); return map; } // Extracts a list of pairs of string views and functions from a sequence object. std::vector>> ExtractKFPairs( PyObject* pyseq) { std::vector>> result; const size_t size = PySequence_Size(pyseq); result.reserve(size); for (size_t i = 0; i < size; i++) { PyObject* pypair = PySequence_GetItem(pyseq, i); if (PySequence_Check(pypair) && PySequence_Size(pypair) >= 2) { PyObject* pykey = PySequence_GetItem(pypair, 0); PyObject* pyfunc = PySequence_GetItem(pypair, 1); if (PyCallable_Check(pyfunc)) { SoftString key(pykey); class Processor final : public tkrzw::DBM::RecordProcessor { public: Processor(PyObject* pyfunc) : pyfunc_(pyfunc) { Py_INCREF(pyfunc_); } ~Processor() { Py_DECREF(pyfunc_); } std::string_view ProcessFull(std::string_view key, std::string_view value) override { PyObject* pyfuncargs = PyTuple_New(2); PyTuple_SET_ITEM(pyfuncargs, 0, CreatePyBytes(key)); PyTuple_SET_ITEM(pyfuncargs, 1, CreatePyBytes(value)); PyObject* pyfuncrv = PyObject_CallObject(pyfunc_, pyfuncargs); std::string_view funcrv = tkrzw::DBM::RecordProcessor::NOOP; if (pyfuncrv != nullptr) { if (pyfuncrv == Py_None) { funcrv = tkrzw::DBM::RecordProcessor::NOOP; } else if (pyfuncrv == Py_False) { funcrv = tkrzw::DBM::RecordProcessor::REMOVE; } else { funcrvstr_ = std::make_unique(pyfuncrv); funcrv = funcrvstr_->Get(); } Py_DECREF(pyfuncrv); } Py_DECREF(pyfuncargs); return funcrv; } std::string_view ProcessEmpty(std::string_view key) override { PyObject* pyfuncargs = PyTuple_New(2); PyTuple_SET_ITEM(pyfuncargs, 0, CreatePyBytes(key)); Py_INCREF(Py_None); PyTuple_SET_ITEM(pyfuncargs, 1, Py_None); PyObject* pyfuncrv = PyObject_CallObject(pyfunc_, pyfuncargs); std::string_view funcrv = tkrzw::DBM::RecordProcessor::NOOP; if (pyfuncrv != nullptr) { if (pyfuncrv == Py_None) { funcrv = tkrzw::DBM::RecordProcessor::NOOP; } else if (pyfuncrv == Py_False) { funcrv = tkrzw::DBM::RecordProcessor::REMOVE; } else { funcrvstr_ = std::make_unique(pyfuncrv); funcrv = funcrvstr_->Get(); } Py_DECREF(pyfuncrv); } Py_DECREF(pyfuncargs); return funcrv; } private: PyObject* pyfunc_; std::unique_ptr funcrvstr_; }; auto proc = std::make_shared(pyfunc); result.emplace_back(std::make_pair(key.Get(), proc)); } Py_DECREF(pyfunc); Py_DECREF(pykey); } Py_DECREF(pypair); } return result; } // Extracts a list of pairs of string views from a sequence object. static std::vector> ExtractSVPairs( PyObject* pyseq, std::vector* placeholder) { std::vector> result; const size_t size = PySequence_Size(pyseq); result.reserve(size); placeholder->reserve(size * 2); for (size_t i = 0; i < size; i++) { PyObject* pypair = PySequence_GetItem(pyseq, i); if (PySequence_Check(pypair) && PySequence_Size(pypair) >= 2) { PyObject* pykey = PySequence_GetItem(pypair, 0); PyObject* pyvalue = PySequence_GetItem(pypair, 1); { SoftString key(pykey); placeholder->emplace_back(std::string(key.Get())); std::string_view key_view = placeholder->back(); std::string_view value_view; if (pyvalue != Py_None) { if (pyvalue == obj_dbm_any_data) { value_view = tkrzw::DBM::ANY_DATA; } else { SoftString value(pyvalue); placeholder->emplace_back(std::string(value.Get())); value_view = placeholder->back(); } } result.emplace_back(std::make_pair(key_view, value_view)); } Py_DECREF(pyvalue); Py_DECREF(pykey); } Py_DECREF(pypair); } return result; } // Defines the module. static bool DefineModule() { static PyModuleDef module_def = {PyModuleDef_HEAD_INIT}; const size_t zoff = offsetof(PyModuleDef, m_name); std::memset((char*)&module_def + zoff, 0, sizeof(module_def) - zoff); module_def.m_name = "tkrzw"; module_def.m_doc = "a set of implementations of DBM"; module_def.m_size = -1; static PyMethodDef methods[] = { {nullptr, nullptr, 0, nullptr}, }; module_def.m_methods = methods; mod_tkrzw = PyModule_Create(&module_def); return true; } // Implementation of Utility.GetMemoryCapacity. static PyObject* utility_GetMemoryCapacity(PyObject* self) { return PyLong_FromLongLong(tkrzw::GetMemoryCapacity()); } // Implementation of Utility.GetMemoryUsage. static PyObject* utility_GetMemoryUsage(PyObject* self) { return PyLong_FromLongLong(tkrzw::GetMemoryUsage()); } // Implementation of Utility.PrimaryHash. static PyObject* utility_PrimaryHash(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydata = PyTuple_GET_ITEM(pyargs, 0); SoftString data(pydata); uint64_t num_buckets = 0; if (argc > 1) { PyObject* pynum = PyTuple_GET_ITEM(pyargs, 1); num_buckets = PyObjToInt(pynum); } if (num_buckets == 0) { num_buckets = tkrzw::UINT64MAX; } return PyLong_FromUnsignedLongLong(tkrzw::PrimaryHash(data.Get(), num_buckets)); } // Implementation of Utility.SecondaryHash. static PyObject* utility_SecondaryHash(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydata = PyTuple_GET_ITEM(pyargs, 0); SoftString data(pydata); uint64_t num_shards = tkrzw::UINT64MAX; if (argc > 1) { PyObject* pynum = PyTuple_GET_ITEM(pyargs, 1); num_shards = PyObjToInt(pynum); } if (num_shards == 0) { num_shards = tkrzw::UINT64MAX; } return PyLong_FromUnsignedLongLong(tkrzw::SecondaryHash(data.Get(), num_shards)); } // Implementation of Utility.EditDistanceLev. static PyObject* utility_EditDistanceLev(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyucsa = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyucsb = PyTuple_GET_ITEM(pyargs, 1); if (!PyUnicode_Check(pyucsa) || PyUnicode_READY(pyucsa) != 0 || !PyUnicode_Check(pyucsb) || PyUnicode_READY(pyucsb) != 0) { ThrowInvalidArguments("not Unicode arguments"); return nullptr; } const std::vector& ucsa = PyUnicodeToUCS4(pyucsa); const std::vector& ucsb = PyUnicodeToUCS4(pyucsb); return PyLong_FromLong(tkrzw::EditDistanceLev>(ucsa, ucsb)); } // Implementation of Utility.SerializeInt. static PyObject* utility_SerializeInt(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pynum = PyTuple_GET_ITEM(pyargs, 0); const int64_t num = PyObjToInt(pynum); const std::string str = tkrzw::IntToStrBigEndian(num, sizeof(int64_t)); return CreatePyBytes(str); } // Implementation of Utility.DeserializeInt. static PyObject* utility_DeserializeInt(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydata = PyTuple_GET_ITEM(pyargs, 0); SoftString data(pydata); const int64_t num = tkrzw::StrToIntBigEndian(data.Get()); return PyLong_FromLongLong(num); } // Implementation of Utility.SerializeFloat. static PyObject* utility_SerializeFloat(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pynum = PyTuple_GET_ITEM(pyargs, 0); const double num = PyObjToDouble(pynum); const std::string str = tkrzw::FloatToStrBigEndian(num, sizeof(double)); return CreatePyBytes(str); } // Implementation of Utility.DeserializeFloat. static PyObject* utility_DeserializeFloat(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydata = PyTuple_GET_ITEM(pyargs, 0); SoftString data(pydata); const double num = tkrzw::StrToFloatBigEndian(data.Get()); return PyFloat_FromDouble(num); } // Defines the Utility class. static bool DefineUtility() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.Utility"; pytype.tp_basicsize = sizeof(PyUtility); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Library utilities."; static PyMethodDef methods[] = { {"GetMemoryCapacity", (PyCFunction)utility_GetMemoryCapacity, METH_CLASS | METH_NOARGS, "Gets the memory capacity of the platform."}, {"GetMemoryUsage", (PyCFunction)utility_GetMemoryUsage, METH_CLASS | METH_NOARGS, "Gets the current memory usage of the process."}, {"PrimaryHash", (PyCFunction)utility_PrimaryHash, METH_CLASS | METH_VARARGS, "Primary hash function for the hash database."}, {"SecondaryHash", (PyCFunction)utility_SecondaryHash, METH_CLASS | METH_VARARGS, "Secondary hash function for sharding."}, {"EditDistanceLev", (PyCFunction)utility_EditDistanceLev, METH_CLASS | METH_VARARGS, "Gets the Levenshtein edit distance of two Unicode strings."}, {"SerializeInt", (PyCFunction)utility_SerializeInt, METH_CLASS | METH_VARARGS, "Serializes an integer into a big-endian binary sequence."}, {"DeserializeInt", (PyCFunction)utility_DeserializeInt, METH_CLASS | METH_VARARGS, "Deserializes a big-endian binary sequence into an integer."}, {"SerializeFloat", (PyCFunction)utility_SerializeFloat, METH_CLASS | METH_VARARGS, "Serializes a floating-point number into a big-endian binary sequence."}, {"DeserializeFloat", (PyCFunction)utility_DeserializeFloat, METH_CLASS | METH_VARARGS, "Deserializes a big-endian binary sequence into a floating-point number."}, {nullptr, nullptr, 0, nullptr}, }; pytype.tp_methods = methods; if (PyType_Ready(&pytype) != 0) return false; cls_utility = (PyObject*)&pytype; Py_INCREF(cls_utility); if (!SetConstStr(cls_utility, "VERSION", tkrzw::PACKAGE_VERSION)) return false; if (!SetConstStr(cls_utility, "OS_NAME", tkrzw::OS_NAME)) return false; if (!SetConstLong(cls_utility, "PAGE_SIZE", tkrzw::PAGE_SIZE)) return false; if (!SetConstLong(cls_utility, "INT32MIN", (int64_t)tkrzw::INT32MIN)) return false; if (!SetConstLong(cls_utility, "INT32MAX", (int64_t)tkrzw::INT32MAX)) return false; if (!SetConstUnsignedLong(cls_utility, "UINT32MAX", (uint64_t)tkrzw::UINT32MAX)) return false; if (!SetConstLong(cls_utility, "INT64MIN", (int64_t)tkrzw::INT64MIN)) return false; if (!SetConstLong(cls_utility, "INT64MAX", (int64_t)tkrzw::INT64MAX)) return false; if (!SetConstUnsignedLong(cls_utility, "UINT64MAX", (uint64_t)tkrzw::UINT64MAX)) return false; if (PyModule_AddObject(mod_tkrzw, "Utility", cls_utility) != 0) return false; return true; } // Implementation of Status.new. static PyObject* status_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyTkStatus* self = (PyTkStatus*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->status = new tkrzw::Status(); return (PyObject*)self; } // Implementation of Status#dealloc. static void status_dealloc(PyTkStatus* self) { delete self->status; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of Status#__init__. static int status_init(PyTkStatus* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 2) { ThrowInvalidArguments("too many arguments"); return -1; } tkrzw::Status::Code code = tkrzw::Status::SUCCESS; if (argc > 0) { PyObject* pycode = PyTuple_GET_ITEM(pyargs, 0); code = (tkrzw::Status::Code)PyLong_AsLong(pycode); } if (argc > 1) { PyObject* pymessage = PyTuple_GET_ITEM(pyargs, 1); SoftString str(pymessage); self->status->Set(code, str.Get()); } else { self->status->Set(code); } return 0; } // Implementation of Status#__repr__. static PyObject* status_repr(PyTkStatus* self) { return CreatePyString(tkrzw::StrCat("status, ">")); } // Implementation of Status#__str__. static PyObject* status_str(PyTkStatus* self) { return CreatePyString(tkrzw::ToString(*self->status)); } // Implementation of Status#__richcmp__. static PyObject* status_richcmp(PyTkStatus* self, PyObject* pyrhs, int op) { bool rv = false; int32_t code = (int32_t)self->status->GetCode(); int32_t rcode = 0; if (PyObject_IsInstance(pyrhs, cls_status)) { PyTkStatus* pyrhs_status = (PyTkStatus*)pyrhs; rcode = (int32_t)pyrhs_status->status->GetCode(); } else if (PyLong_Check(pyrhs)) { rcode = (int32_t)PyLong_AsLong(pyrhs); } else { rcode = tkrzw::INT32MAX; } switch (op) { case Py_LT: rv = code < rcode; break; case Py_LE: rv = code <= rcode; break; case Py_EQ: rv = code == rcode; break; case Py_NE: rv = code != rcode; break; case Py_GT: rv = code > rcode; break; case Py_GE: rv = code >= rcode; break; default: rv = false; break; } if (rv) { Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of Status#Set. static PyObject* status_Set(PyTkStatus* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 2) { ThrowInvalidArguments("too many arguments"); return nullptr; } tkrzw::Status::Code code = tkrzw::Status::SUCCESS; if (argc > 0) { PyObject* pycode = PyTuple_GET_ITEM(pyargs, 0); code = (tkrzw::Status::Code)PyLong_AsLong(pycode); } if (argc > 1) { PyObject* pymessage = PyTuple_GET_ITEM(pyargs, 1); SoftString str(pymessage); self->status->Set(code, str.Get()); } else { self->status->Set(code); } Py_RETURN_NONE; } // Implementation of Status#Join. static PyObject* status_Join(PyTkStatus* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyrht = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pyrht, cls_status)) { ThrowInvalidArguments("the argument is not a Status"); return nullptr; } PyTkStatus* rht = (PyTkStatus*)pyrht; (*self->status) |= (*rht->status); Py_RETURN_NONE; } // Implementation of Status#GetCode. static PyObject* status_GetCode(PyTkStatus* self) { return PyLong_FromLongLong(self->status->GetCode()); } // Implementation of Status#GetMessage. static PyObject* status_GetMessage(PyTkStatus* self) { return PyUnicode_FromString(self->status->GetMessage().c_str()); } // Implementation of Status#IsOK. static PyObject* status_IsOK(PyTkStatus* self) { if (*self->status == tkrzw::Status::SUCCESS) { Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of Status#OrDie. static PyObject* status_OrDie(PyTkStatus* self) { if (*self->status != tkrzw::Status::SUCCESS) { ThrowStatusException(*self->status); return nullptr; } Py_RETURN_NONE; } // Implementation of Status.CodeName. static PyObject* status_CodeName(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pycode = PyTuple_GET_ITEM(pyargs, 0); const tkrzw::Status::Code code = (tkrzw::Status::Code)PyLong_AsLong(pycode); return CreatePyString(tkrzw::ToString(tkrzw::Status::CodeName(code))); } // Defines the Status class. static bool DefineStatus() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.Status"; pytype.tp_basicsize = sizeof(PyTkStatus); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Status of operations."; pytype.tp_new = status_new; pytype.tp_dealloc = (destructor)status_dealloc; pytype.tp_init = (initproc)status_init; pytype.tp_repr = (unaryfunc)status_repr; pytype.tp_str = (unaryfunc)status_str; pytype.tp_richcompare = (richcmpfunc)status_richcmp; static PyMethodDef methods[] = { {"Set", (PyCFunction)status_Set, METH_VARARGS, "Set the code and the message."}, {"Join", (PyCFunction)status_Join, METH_VARARGS, "Assigns the internal state only if the current state is success."}, {"GetCode", (PyCFunction)status_GetCode, METH_NOARGS, "Gets the status code.."}, {"GetMessage", (PyCFunction)status_GetMessage, METH_NOARGS, "Gets the status message."}, {"IsOK", (PyCFunction)status_IsOK, METH_NOARGS, "Returns true if the status is success."}, {"OrDie", (PyCFunction)status_OrDie, METH_NOARGS, "Raises a runtime error if the status is not success."}, {"CodeName", (PyCFunction)status_CodeName, METH_CLASS | METH_VARARGS, "Gets the string name of a status code."}, {nullptr, nullptr, 0, nullptr}, }; pytype.tp_methods = methods; if (PyType_Ready(&pytype) != 0) return false; cls_status = (PyObject*)&pytype; Py_INCREF(cls_status); if (!SetConstLong(cls_status, "SUCCESS", (int64_t)tkrzw::Status::SUCCESS)) return false; if (!SetConstLong(cls_status, "UNKNOWN_ERROR", (int64_t)tkrzw::Status::UNKNOWN_ERROR)) return false; if (!SetConstLong(cls_status, "SYSTEM_ERROR", (int64_t)tkrzw::Status::SYSTEM_ERROR)) return false; if (!SetConstLong(cls_status, "NOT_IMPLEMENTED_ERROR", (int64_t)tkrzw::Status::NOT_IMPLEMENTED_ERROR)) return false; if (!SetConstLong(cls_status, "PRECONDITION_ERROR", (int64_t)tkrzw::Status::PRECONDITION_ERROR)) return false; if (!SetConstLong(cls_status, "INVALID_ARGUMENT_ERROR", (int64_t)tkrzw::Status::INVALID_ARGUMENT_ERROR)) return false; if (!SetConstLong(cls_status, "CANCELED_ERROR", (int64_t)tkrzw::Status::CANCELED_ERROR)) return false; if (!SetConstLong(cls_status, "NOT_FOUND_ERROR", (int64_t)tkrzw::Status::NOT_FOUND_ERROR)) return false; if (!SetConstLong(cls_status, "PERMISSION_ERROR", (int64_t)tkrzw::Status::PERMISSION_ERROR)) return false; if (!SetConstLong(cls_status, "INFEASIBLE_ERROR", (int64_t)tkrzw::Status::INFEASIBLE_ERROR)) return false; if (!SetConstLong(cls_status, "DUPLICATION_ERROR", (int64_t)tkrzw::Status::DUPLICATION_ERROR)) return false; if (!SetConstLong(cls_status, "BROKEN_DATA_ERROR", (int64_t)tkrzw::Status::BROKEN_DATA_ERROR)) return false; if (!SetConstLong(cls_status, "NETWORK_ERROR", (int64_t)tkrzw::Status::NETWORK_ERROR)) return false; if (!SetConstLong(cls_status, "APPLICATION_ERROR", (int64_t)tkrzw::Status::APPLICATION_ERROR)) return false; if (PyModule_AddObject(mod_tkrzw, "Status", cls_status) != 0) return false; return true; } // Implementation of StatusException.new. static PyObject* expt_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyException* self = (PyException*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->pystatus = nullptr; return (PyObject*)self; } // Implementation of StatusException#dealloc. static void expt_dealloc(PyException* self) { if (self->pystatus) Py_DECREF(self->pystatus); Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of StatusException#__init__. static int expt_init(PyException* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return -1; } PyObject* pystatus = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("the argument is not a status"); return -1; } Py_INCREF(pystatus); self->pystatus = pystatus; return 0; } // Implementation of StatusException#__repr__. static PyObject* expt_repr(PyException* self) { const tkrzw::Status* status = ((PyTkStatus*)self->pystatus)->status; return CreatePyString(tkrzw::StrCat("")); } // Implementation of StatusException#__str__. static PyObject* expt_str(PyException* self) { const tkrzw::Status* status = ((PyTkStatus*)self->pystatus)->status; return CreatePyString(tkrzw::ToString(*status)); } // Implementation of StatusException#GetStatus. static PyObject* expt_GetStatus(PyException* self) { Py_INCREF(self->pystatus); return self->pystatus; } // Defines the StatusException class. static bool DefineStatusException() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.StatusException"; pytype.tp_basicsize = sizeof(PyException); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Exception to convey the status of operations."; pytype.tp_new = expt_new; pytype.tp_dealloc = (destructor)expt_dealloc; pytype.tp_init = (initproc)expt_init; pytype.tp_repr = (unaryfunc)expt_repr; pytype.tp_str = (unaryfunc)expt_str; static PyMethodDef methods[] = { {"GetStatus", (PyCFunction)expt_GetStatus, METH_NOARGS, "Get the status object." }, {nullptr, nullptr, 0, nullptr} }; pytype.tp_methods = methods; pytype.tp_base = (PyTypeObject*)PyExc_RuntimeError; if (PyType_Ready(&pytype) != 0) return false; cls_expt = (PyObject*)&pytype; Py_INCREF(cls_expt); if (PyModule_AddObject(mod_tkrzw, "StatusException", cls_expt) != 0) return false; return true; } // Implementation of Future.new. static PyObject* future_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyFuture* self = (PyFuture*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->future = nullptr; self->concurrent = false; self->is_str = false; return (PyObject*)self; } // Implementation of Future#dealloc. static void future_dealloc(PyFuture* self) { delete self->future; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of Future#__init__. static int future_init(PyFuture* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 0) { ThrowInvalidArguments("too many arguments"); return -1; } ThrowStatusException(tkrzw::Status(tkrzw::Status::NOT_IMPLEMENTED_ERROR)); return -1; } // Implementation of Future#__repr__. static PyObject* future_repr(PyFuture* self) { const std::string& str = tkrzw::SPrintF("", (void*)self->future); return CreatePyString(str); } // Implementation of Future#__str__. static PyObject* future_str(PyFuture* self) { const std::string& str = tkrzw::SPrintF("Future:%p", (void*)self->future); return CreatePyString(str); } // Implementation of Future#__iter__. static PyObject* future_iter(PyFuture* self) { Py_INCREF(self); return (PyObject*)self; } // Implementation of Future#__next__. static PyObject* future_iternext(PyFuture* self) { PyErr_SetString(PyExc_StopIteration, "end of iteration"); return nullptr; } // Implementation of Future#__await__. static PyObject* future_await(PyFuture* self) { { NativeLock lock(self->concurrent); self->future->Wait(); } self->concurrent = false; Py_INCREF(self); return (PyObject*)self; } // Implementation of Future#Wait. static PyObject* future_Wait(PyFuture* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments("too many arguments"); return nullptr; } const double timeout = argc > 0 ? PyObjToDouble(PyTuple_GET_ITEM(pyargs, 0)) : -1.0; bool ok = false; { NativeLock lock(self->concurrent); ok = self->future->Wait(timeout); } if (ok) { self->concurrent = false; Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of Future#Get. static PyObject* future_Get(PyFuture* self) { const auto& type = self->future->GetExtraType(); if (type == typeid(tkrzw::Status)) { NativeLock lock(self->concurrent); tkrzw::Status status = self->future->Get(); lock.Release(); delete self->future; self->future = nullptr; return CreatePyTkStatusMove(std::move(status)); } if (type == typeid(std::pair)) { NativeLock lock(self->concurrent); const auto& result = self->future->GetString(); lock.Release(); delete self->future; self->future = nullptr; PyObject* pyrv = PyTuple_New(2); PyTuple_SET_ITEM(pyrv, 0, CreatePyTkStatus(std::move(result.first))); if (self->is_str) { PyTuple_SET_ITEM(pyrv, 1, CreatePyString(result.second)); } else { PyTuple_SET_ITEM(pyrv, 1, CreatePyBytes(result.second)); } return pyrv; } if (type == typeid(std::pair>)) { NativeLock lock(self->concurrent); const auto& result = self->future->GetStringPair(); lock.Release(); delete self->future; self->future = nullptr; PyObject* pyrv = PyTuple_New(3); PyTuple_SET_ITEM(pyrv, 0, CreatePyTkStatus(std::move(result.first))); if (self->is_str) { PyTuple_SET_ITEM(pyrv, 1, CreatePyString(result.second.first)); PyTuple_SET_ITEM(pyrv, 2, CreatePyString(result.second.second)); } else { PyTuple_SET_ITEM(pyrv, 1, CreatePyBytes(result.second.first)); PyTuple_SET_ITEM(pyrv, 2, CreatePyBytes(result.second.second)); } return pyrv; } if (type == typeid(std::pair>)) { NativeLock lock(self->concurrent); const auto& result = self->future->GetStringVector(); lock.Release(); delete self->future; self->future = nullptr; PyObject* pyrv = PyTuple_New(2); PyTuple_SET_ITEM(pyrv, 0, CreatePyTkStatus(std::move(result.first))); PyObject* pylist = PyTuple_New(result.second.size()); for (size_t i = 0; i < result.second.size(); i++) { if (self->is_str) { PyTuple_SET_ITEM(pylist, i, CreatePyString(result.second[i])); } else { PyTuple_SET_ITEM(pylist, i, CreatePyBytes(result.second[i])); } } PyTuple_SET_ITEM(pyrv, 1, pylist); return pyrv; } if (type == typeid(std::pair>)) { NativeLock lock(self->concurrent); const auto& result = self->future->GetStringMap(); lock.Release(); delete self->future; self->future = nullptr; PyObject* pyrv = PyTuple_New(2); PyTuple_SET_ITEM(pyrv, 0, CreatePyTkStatus(std::move(result.first))); PyObject* pydict = PyDict_New(); for (const auto& rec : result.second) { if (self->is_str) { PyObject* pykey = CreatePyString(rec.first); PyObject* pyvalue = CreatePyString(rec.second); PyDict_SetItem(pydict, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); } else { PyObject* pykey = CreatePyBytes(rec.first); PyObject* pyvalue = CreatePyBytes(rec.second); PyDict_SetItem(pydict, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); } } PyTuple_SET_ITEM(pyrv, 1, pydict); return pyrv; } if (type == typeid(std::pair)) { NativeLock lock(self->concurrent); const auto& result = self->future->GetInteger(); lock.Release(); delete self->future; self->future = nullptr; PyObject* pyrv = PyTuple_New(2); PyTuple_SET_ITEM(pyrv, 0, CreatePyTkStatus(std::move(result.first))); PyTuple_SET_ITEM(pyrv, 1, PyLong_FromLongLong(result.second)); return pyrv; } ThrowStatusException(tkrzw::Status(tkrzw::Status::NOT_IMPLEMENTED_ERROR)); return nullptr; } // Defines the Future class. static bool DefineFuture() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.Future"; pytype.tp_basicsize = sizeof(PyFuture); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Future to monitor the result of asynchronous operations."; pytype.tp_new = future_new; pytype.tp_dealloc = (destructor)future_dealloc; pytype.tp_init = (initproc)future_init; pytype.tp_repr = (unaryfunc)future_repr; pytype.tp_str = (unaryfunc)future_str; static PyMethodDef methods[] = { {"Wait", (PyCFunction)future_Wait, METH_VARARGS, "Waits for the operation to be done."}, {"Get", (PyCFunction)future_Get, METH_NOARGS, "Waits for the operation to be done and gets the result status." }, {nullptr, nullptr, 0, nullptr} }; pytype.tp_methods = methods; static PyAsyncMethods async_methods; std::memset(&async_methods, 0, sizeof(async_methods)); async_methods.am_await = (unaryfunc)future_await; pytype.tp_as_async = &async_methods; static PyMappingMethods map_methods; std::memset(&map_methods, 0, sizeof(map_methods)); pytype.tp_iter = (getiterfunc)future_iter; pytype.tp_iternext = (iternextfunc)future_iternext; pytype.tp_as_mapping = &map_methods; if (PyType_Ready(&pytype) != 0) return false; cls_future = (PyObject*)&pytype; Py_INCREF(cls_future); if (PyModule_AddObject(mod_tkrzw, "Future", cls_future) != 0) return false; return true; } // Implementation of DBM.new. static PyObject* dbm_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyDBM* self = (PyDBM*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->dbm = nullptr; self->concurrent = false; return (PyObject*)self; } // Implementation of DBM#dealloc. static void dbm_dealloc(PyDBM* self) { delete self->dbm; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of DBM#__init__. static int dbm_init(PyDBM* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 0) { ThrowInvalidArguments("too many arguments"); return -1; } return 0; } // Implementation of DBM#__repr__. static PyObject* dbm_repr(PyDBM* self) { std::string class_name = "unknown"; std::string path = "-"; int64_t num_records = -1; if (self->dbm != nullptr) { NativeLock lock(self->concurrent); for (const auto& rec : self->dbm->Inspect()) { if (rec.first == "class") { class_name = rec.second; } else if (rec.first == "path") { path = rec.second; } } num_records = self->dbm->CountSimple(); } const std::string& str = tkrzw::StrCat( ""); return CreatePyString(str); } // Implementation of DBM#__str__. static PyObject* dbm_str(PyDBM* self) { std::string class_name = "unknown"; std::string path = "-"; int64_t num_records = -1; if (self->dbm != nullptr) { NativeLock lock(self->concurrent); for (const auto& rec : self->dbm->Inspect()) { if (rec.first == "class") { class_name = rec.second; } else if (rec.first == "path") { path = rec.second; } } num_records = self->dbm->CountSimple(); } const std::string& str = tkrzw::StrCat( class_name, ":", tkrzw::StrEscapeC(path, true), ":", num_records); return CreatePyString(str); } // Implementation of DBM#Open. static PyObject* dbm_Open(PyDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->dbm != nullptr) { ThrowInvalidArguments("opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pypath = PyTuple_GET_ITEM(pyargs, 0); PyObject* pywritable = PyTuple_GET_ITEM(pyargs, 1); SoftString path(pypath); const bool writable = PyObject_IsTrue(pywritable); int32_t num_shards = -1; bool concurrent = false; int32_t open_options = 0; std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); num_shards = tkrzw::StrToInt(tkrzw::SearchMap(params, "num_shards", "-1")); if (tkrzw::StrToBool(tkrzw::SearchMap(params, "concurrent", "false"))) { concurrent = true; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "truncate", "false"))) { open_options |= tkrzw::File::OPEN_TRUNCATE; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_create", "false"))) { open_options |= tkrzw::File::OPEN_NO_CREATE; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_wait", "false"))) { open_options |= tkrzw::File::OPEN_NO_WAIT; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_lock", "false"))) { open_options |= tkrzw::File::OPEN_NO_LOCK; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "sync_hard", "false"))) { open_options |= tkrzw::File::OPEN_SYNC_HARD; } params.erase("concurrent"); params.erase("truncate"); params.erase("no_create"); params.erase("no_wait"); params.erase("no_lock"); params.erase("sync_hard"); } if (num_shards >= 0) { self->dbm = new tkrzw::ShardDBM(); } else { self->dbm = new tkrzw::PolyDBM(); } self->concurrent = concurrent; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->OpenAdvanced(std::string(path.Get()), writable, open_options, params); } if (status != tkrzw::Status::SUCCESS) { delete self->dbm; self->dbm = nullptr; } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Close. static PyObject* dbm_Close(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Close(); } delete self->dbm; self->dbm = nullptr; return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Process. static PyObject* dbm_Process(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } if (self->concurrent) { return CreatePyTkStatusMove(tkrzw::Status( tkrzw::Status::PRECONDITION_ERROR, "the concurrent mode is not supported")); } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 3) { ThrowInvalidArguments(argc < 3 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyfunc = PyTuple_GET_ITEM(pyargs, 1); const bool writable = PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 2)); if (!PyCallable_Check(pyfunc)) { ThrowInvalidArguments("non callable is given"); return nullptr; } SoftString key(pykey); std::unique_ptr funcrvstr; auto func = [&](std::string_view k, std::string_view v) -> std::string_view { PyObject* pyfuncargs = PyTuple_New(2); PyTuple_SET_ITEM(pyfuncargs, 0, CreatePyBytes(k)); if (v.data() == tkrzw::DBM::RecordProcessor::NOOP.data()) { Py_INCREF(Py_None); PyTuple_SET_ITEM(pyfuncargs, 1, Py_None); } else { PyTuple_SET_ITEM(pyfuncargs, 1, CreatePyBytes(v)); } PyObject* pyfuncrv = PyObject_CallObject(pyfunc, pyfuncargs); std::string_view funcrv = tkrzw::DBM::RecordProcessor::NOOP; if (pyfuncrv != nullptr) { if (pyfuncrv == Py_None) { funcrv = tkrzw::DBM::RecordProcessor::NOOP; } else if (pyfuncrv == Py_False) { funcrv = tkrzw::DBM::RecordProcessor::REMOVE; } else { funcrvstr = std::make_unique(pyfuncrv); funcrv = funcrvstr->Get(); } Py_DECREF(pyfuncrv); } Py_DECREF(pyfuncargs); return funcrv; }; tkrzw::Status status = self->dbm->Process(key.Get(), func, writable); return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Get. static PyObject* dbm_Get(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); PyObject* pystatus = nullptr; if (argc > 1) { pystatus = PyTuple_GET_ITEM(pyargs, 1); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Get(key.Get(), &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return CreatePyBytes(value); } // Implementation of DBM#GetStr. static PyObject* dbm_GetStr(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); PyObject* pystatus = nullptr; if (argc > 1) { pystatus = PyTuple_GET_ITEM(pyargs, 1); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Get(key.Get(), &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return CreatePyString(value); } // Implementation of DBM#GetMulti. static PyObject* dbm_GetMulti(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); std::vector keys; for (int32_t i = 0; i < argc; i++) { PyObject* pykey = PyTuple_GET_ITEM(pyargs, i); SoftString key(pykey); keys.emplace_back(std::string(key.Get())); } std::vector key_views(keys.begin(), keys.end()); std::map records; { NativeLock lock(self->concurrent); self->dbm->GetMulti(key_views, &records); } PyObject* pyrv = PyDict_New(); for (const auto& rec : records) { PyObject* pyname = CreatePyBytes(rec.first); PyObject* pyvalue = CreatePyBytes(rec.second); PyDict_SetItem(pyrv, pyname, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pyname); } return pyrv; } // Implementation of DBM#GetMultiStr. static PyObject* dbm_GetMultiStr(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); std::vector keys; for (int32_t i = 0; i < argc; i++) { PyObject* pykey = PyTuple_GET_ITEM(pyargs, i); SoftString key(pykey); keys.emplace_back(std::string(key.Get())); } std::vector key_views(keys.begin(), keys.end()); std::map records; { NativeLock lock(self->concurrent); self->dbm->GetMulti(key_views, &records); } PyObject* pyrv = PyDict_New(); for (const auto& rec : records) { PyObject* pyname = CreatePyString(rec.first); PyObject* pyvalue = CreatePyString(rec.second); PyDict_SetItem(pyrv, pyname, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pyname); } return pyrv; } // Implementation of DBM#Set. static PyObject* dbm_Set(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); const bool overwrite = argc > 2 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 2)) : true; SoftString key(pykey); SoftString value(pyvalue); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Set(key.Get(), value.Get(), overwrite); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#SetMulti. static PyObject* dbm_SetMulti(PyDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments("too many arguments"); return nullptr; } PyObject* pyoverwrite = argc > 0 ? PyTuple_GET_ITEM(pyargs, 0) : Py_True; const bool overwrite = PyObject_IsTrue(pyoverwrite); std::map records; if (pykwds != nullptr) { records = MapKeywords(pykwds); } std::map record_views; for (const auto& record : records) { record_views.emplace(std::make_pair( std::string_view(record.first), std::string_view(record.second))); } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->SetMulti(record_views, overwrite); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#SetAndGet. static PyObject* dbm_SetAndGet(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); const bool overwrite = argc > 2 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 2)) : true; SoftString key(pykey); SoftString value(pyvalue); tkrzw::Status impl_status(tkrzw::Status::SUCCESS); std::string old_value; bool hit = false; class Processor final : public tkrzw::DBM::RecordProcessor { public: Processor(tkrzw::Status* status, std::string_view value, bool overwrite, std::string* old_value, bool* hit) : status_(status), value_(value), overwrite_(overwrite), old_value_(old_value), hit_(hit) {} std::string_view ProcessFull(std::string_view key, std::string_view value) override { *old_value_ = value; *hit_ = true; if (overwrite_) { return value_; } status_->Set(tkrzw::Status::DUPLICATION_ERROR); return NOOP; } std::string_view ProcessEmpty(std::string_view key) override { return value_; } private: tkrzw::Status* status_; std::string_view value_; bool overwrite_; std::string* old_value_; bool* hit_; }; Processor proc(&impl_status, value.Get(), overwrite, &old_value, &hit); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Process(key.Get(), &proc, true); } status |= impl_status; PyObject* pytuple = PyTuple_New(2); PyTuple_SET_ITEM(pytuple, 0, CreatePyTkStatusMove(std::move(status))); if (hit) { PyObject* pyold_value = PyUnicode_Check(pyvalue) ? CreatePyString(old_value) : CreatePyBytes(old_value); PyTuple_SET_ITEM(pytuple, 1, pyold_value); } else { Py_INCREF(Py_None); PyTuple_SET_ITEM(pytuple, 1, Py_None); } return pytuple; } // Implementation of DBM#Remove. static PyObject* dbm_Remove(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Remove(key.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#RemoveMulti. static PyObject* dbm_RemoveMulti(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); std::vector keys; for (int32_t i = 0; i < argc; i++) { PyObject* pykey = PyTuple_GET_ITEM(pyargs, i); SoftString key(pykey); keys.emplace_back(std::string(key.Get())); } std::vector key_views(keys.begin(), keys.end()); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->RemoveMulti(key_views); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#RemoveAndGet. static PyObject* dbm_RemoveAndGet(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); tkrzw::Status impl_status(tkrzw::Status::SUCCESS); std::string old_value; class Processor final : public tkrzw::DBM::RecordProcessor { public: Processor(tkrzw::Status* status, std::string* old_value) : status_(status), old_value_(old_value) {} std::string_view ProcessFull(std::string_view key, std::string_view value) override { *old_value_ = value; return REMOVE; } std::string_view ProcessEmpty(std::string_view key) override { status_->Set(tkrzw::Status::NOT_FOUND_ERROR); return NOOP; } private: tkrzw::Status* status_; std::string* old_value_; }; Processor proc(&impl_status, &old_value); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Process(key.Get(), &proc, true); } status |= impl_status; PyObject* pytuple = PyTuple_New(2); const bool success = status == tkrzw::Status::SUCCESS; PyTuple_SET_ITEM(pytuple, 0, CreatePyTkStatusMove(std::move(status))); if (success) { PyObject* pyold_value = PyUnicode_Check(pykey) ? CreatePyString(old_value) : CreatePyBytes(old_value); PyTuple_SET_ITEM(pytuple, 1, pyold_value); } else { Py_INCREF(Py_None); PyTuple_SET_ITEM(pytuple, 1, Py_None); } return pytuple; } // Implementation of DBM#Append. static PyObject* dbm_Append(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); PyObject* pydelim = argc > 2 ? PyTuple_GET_ITEM(pyargs, 2) : nullptr; SoftString key(pykey); SoftString value(pyvalue); SoftString delim(pydelim == nullptr ? Py_None : pydelim); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Append(key.Get(), value.Get(), delim.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#AppendMulti. static PyObject* dbm_AppendMulti(PyDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments("too many arguments"); return nullptr; } PyObject* pydelim = argc > 0 ? PyTuple_GET_ITEM(pyargs, 0) : nullptr; SoftString delim(pydelim == nullptr ? Py_None : pydelim); std::map records; if (pykwds != nullptr) { records = MapKeywords(pykwds); } std::map record_views; for (const auto& record : records) { record_views.emplace(std::make_pair( std::string_view(record.first), std::string_view(record.second))); } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->AppendMulti(record_views, delim.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#CompareExchange. static PyObject* dbm_CompareExchange(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 3) { ThrowInvalidArguments(argc < 3 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyexpected = PyTuple_GET_ITEM(pyargs, 1); PyObject* pydesired = PyTuple_GET_ITEM(pyargs, 2); SoftString key(pykey); std::unique_ptr expected; std::string_view expected_view; if (pyexpected != Py_None) { if (pyexpected == obj_dbm_any_data) { expected_view = tkrzw::DBM::ANY_DATA; } else { expected = std::make_unique(pyexpected); expected_view = expected->Get(); } } std::unique_ptr desired; std::string_view desired_view; if (pydesired != Py_None) { if (pydesired == obj_dbm_any_data) { desired_view = tkrzw::DBM::ANY_DATA; } else { desired = std::make_unique(pydesired); desired_view = desired->Get(); } } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->CompareExchange(key.Get(), expected_view, desired_view); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#CompareExchangeAndGet. static PyObject* dbm_CompareExchangeAndGet(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 3) { ThrowInvalidArguments(argc < 3 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyexpected = PyTuple_GET_ITEM(pyargs, 1); PyObject* pydesired = PyTuple_GET_ITEM(pyargs, 2); SoftString key(pykey); std::unique_ptr expected; std::string_view expected_view; if (pyexpected != Py_None) { if (pyexpected == obj_dbm_any_data) { expected_view = tkrzw::DBM::ANY_DATA; } else { expected = std::make_unique(pyexpected); expected_view = expected->Get(); } } std::unique_ptr desired; std::string_view desired_view; if (pydesired != Py_None) { if (pydesired == obj_dbm_any_data) { desired_view = tkrzw::DBM::ANY_DATA; } else { desired = std::make_unique(pydesired); desired_view = desired->Get(); } } tkrzw::Status status(tkrzw::Status::SUCCESS); std::string actual; bool found = false; { NativeLock lock(self->concurrent); status = self->dbm->CompareExchange(key.Get(), expected_view, desired_view, &actual, &found); } PyObject* pytuple = PyTuple_New(2); PyTuple_SET_ITEM(pytuple, 0, CreatePyTkStatusMove(std::move(status))); if (found) { PyObject* pyactual = PyUnicode_Check(pyexpected) || PyUnicode_Check(pydesired) ? CreatePyString(actual) : CreatePyBytes(actual); PyTuple_SET_ITEM(pytuple, 1, pyactual); } else { Py_INCREF(Py_None); PyTuple_SET_ITEM(pytuple, 1, Py_None); } return pytuple; } // Implementation of DBM#Increment. static PyObject* dbm_Increment(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 4) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); int64_t inc = 1; if (argc > 1) { PyObject* pyinc = PyTuple_GET_ITEM(pyargs, 1); inc = PyObjToInt(pyinc); } int64_t init = 0; if (argc > 2) { PyObject* pyinit = PyTuple_GET_ITEM(pyargs, 2); init = PyObjToInt(pyinit); } PyObject* pystatus = nullptr; if (argc > 3) { pystatus = PyTuple_GET_ITEM(pyargs, 3); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } tkrzw::Status status(tkrzw::Status::SUCCESS); int64_t current = 0; { NativeLock lock(self->concurrent); status = self->dbm->Increment(key.Get(), inc, ¤t, init); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { return PyLong_FromLongLong(current); } Py_RETURN_NONE; } // Implementation of DBM#ProcessMulti. static PyObject* dbm_ProcessMulti(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } if (self->concurrent) { return CreatePyTkStatusMove(tkrzw::Status( tkrzw::Status::PRECONDITION_ERROR, "the concurrent mode is not supported")); } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykfpairs = PyTuple_GET_ITEM(pyargs, 0); const bool writable = PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 1)); if (!PySequence_Check(pykfpairs)) { ThrowInvalidArguments("parameters must be sequences of tuples and strings and functions"); return nullptr; } const auto& kfpairs_ph = ExtractKFPairs(pykfpairs); std::vector> kfpairs; kfpairs.reserve(kfpairs_ph.size()); for (const auto& key_proc : kfpairs_ph) { auto kfpair = std::make_pair(std::string_view(key_proc.first), key_proc.second.get()); kfpairs.emplace_back(std::move(kfpair)); } tkrzw::Status status = self->dbm->ProcessMulti(kfpairs, writable); return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#CompareExchangeMulti. static PyObject* dbm_CompareExchangeMulti(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyexpected = PyTuple_GET_ITEM(pyargs, 0); PyObject* pydesired = PyTuple_GET_ITEM(pyargs, 1); if (!PySequence_Check(pyexpected) || !PySequence_Check(pydesired)) { ThrowInvalidArguments("parameters must be sequences of strings"); return nullptr; } std::vector expected_ph; const auto& expected = ExtractSVPairs(pyexpected, &expected_ph); std::vector desired_ph; const auto& desired = ExtractSVPairs(pydesired, &desired_ph); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->CompareExchangeMulti(expected, desired); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Rekey. static PyObject* dbm_Rekey(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 4) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyold_key = PyTuple_GET_ITEM(pyargs, 0); PyObject* pynew_key = PyTuple_GET_ITEM(pyargs, 1); const bool overwrite = argc > 2 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 2)) : true; const bool copying = argc > 3 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 3)) : false; SoftString old_key(pyold_key); SoftString new_key(pynew_key); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Rekey(old_key.Get(), new_key.Get(), overwrite, copying); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#PopFirst. static PyObject* dbm_PopFirst(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->PopFirst(&key, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyBytes(key); PyObject* pyvalue = CreatePyBytes(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of DBM#PopFirstStr. static PyObject* dbm_PopFirstStr(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->PopFirst(&key, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyString(key); PyObject* pyvalue = CreatePyString(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of DBM#PushLast. static PyObject* dbm_PushLast(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 0); const double wtime = argc > 1 ? PyObjToDouble(PyTuple_GET_ITEM(pyargs, 1)) : -1; SoftString value(pyvalue); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->PushLast(value.Get(), wtime); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#ProcessEach. static PyObject* dbm_ProcessEach(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } if (self->concurrent) { return CreatePyTkStatusMove(tkrzw::Status( tkrzw::Status::PRECONDITION_ERROR, "the concurrent mode is not supported")); } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyfunc = PyTuple_GET_ITEM(pyargs, 0); const bool writable = PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 1)); if (!PyCallable_Check(pyfunc)) { ThrowInvalidArguments("non callable is given"); return nullptr; } std::unique_ptr funcrvstr; auto func = [&](std::string_view k, std::string_view v) -> std::string_view { PyObject* pyfuncargs = PyTuple_New(2); if (k.data() == tkrzw::DBM::RecordProcessor::NOOP.data()) { Py_INCREF(Py_None); PyTuple_SET_ITEM(pyfuncargs, 0, Py_None); } else { PyTuple_SET_ITEM(pyfuncargs, 0, CreatePyBytes(k)); } if (v.data() == tkrzw::DBM::RecordProcessor::NOOP.data()) { Py_INCREF(Py_None); PyTuple_SET_ITEM(pyfuncargs, 1, Py_None); } else { PyTuple_SET_ITEM(pyfuncargs, 1, CreatePyBytes(v)); } PyObject* pyfuncrv = PyObject_CallObject(pyfunc, pyfuncargs); std::string_view funcrv = tkrzw::DBM::RecordProcessor::NOOP; if (pyfuncrv != nullptr) { if (pyfuncrv == Py_None) { funcrv = tkrzw::DBM::RecordProcessor::NOOP; } else if (pyfuncrv == Py_False) { funcrv = tkrzw::DBM::RecordProcessor::REMOVE; } else { funcrvstr = std::make_unique(pyfuncrv); funcrv = funcrvstr->Get(); } Py_DECREF(pyfuncrv); } Py_DECREF(pyfuncargs); return funcrv; }; tkrzw::Status status = self->dbm->ProcessEach(func, writable); return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Count. static PyObject* dbm_Count(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } int64_t count = -1; { NativeLock lock(self->concurrent); count = self->dbm->CountSimple(); } if (count >= 0) { return PyLong_FromLongLong(count); } Py_RETURN_NONE; } // Implementation of DBM#GetFileSize. static PyObject* dbm_GetFileSize(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } int64_t file_size = -1; { NativeLock lock(self->concurrent); file_size = self->dbm->GetFileSizeSimple(); } if (file_size >= 0) { return PyLong_FromLongLong(file_size); } Py_RETURN_NONE; } // Implementation of DBM#GetFilePath. static PyObject* dbm_GetFilePath(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } std::string path; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->GetFilePath(&path); } if (status == tkrzw::Status::SUCCESS) { return CreatePyString(path); } Py_RETURN_NONE; } // Implementation of DBM#GetTimestamp. static PyObject* dbm_GetTimestamp(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } double timestamp = -1; { NativeLock lock(self->concurrent); timestamp = self->dbm->GetTimestampSimple(); } if (timestamp >= 0) { return PyFloat_FromDouble(timestamp); } Py_RETURN_NONE; } // Implementation of DBM#Clear. static PyObject* dbm_Clear(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Clear(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Rebuild. static PyObject* dbm_Rebuild(PyDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 0) { ThrowInvalidArguments("too many arguments"); return nullptr; } std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->RebuildAdvanced(params); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#ShouldBeRebuilt. static PyObject* dbm_ShouldBeRebuilt(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } bool tobe = false; { NativeLock lock(self->concurrent); tobe = self->dbm->ShouldBeRebuiltSimple(); } return PyBool_FromLong(tobe); } // Implementation of DBM#Synchronize. static PyObject* dbm_Synchronize(PyDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyhard = PyTuple_GET_ITEM(pyargs, 0); const bool hard = PyObject_IsTrue(pyhard); std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->SynchronizeAdvanced(hard, nullptr, params); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#CopyFileData. static PyObject* dbm_CopyFileData(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydest = PyTuple_GET_ITEM(pyargs, 0); bool sync_hard = false; if (argc > 1) { PyObject* pysync_hard = PyTuple_GET_ITEM(pyargs, 1); sync_hard = PyObject_IsTrue(pysync_hard); } SoftString dest(pydest); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->CopyFileData(std::string(dest.Get()), sync_hard); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Export. static PyObject* dbm_Export(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydest = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydest, cls_dbm)) { ThrowInvalidArguments("the argument is not a DBM"); return nullptr; } PyDBM* dest = (PyDBM*)pydest; if (dest->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Export(dest->dbm); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#ExportToFlatRecords. static PyObject* dbm_ExportToFlatRecords(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydest_file = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydest_file, cls_file)) { ThrowInvalidArguments("the argument is not a File"); return nullptr; } PyFile* dest_file = (PyFile*)pydest_file; if (dest_file->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = tkrzw::ExportDBMToFlatRecords(self->dbm, dest_file->file); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#ImportFromFlatRecords. static PyObject* dbm_ImportFromFlatRecords(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pysrc_file = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pysrc_file, cls_file)) { ThrowInvalidArguments("the argument is not a File"); return nullptr; } PyFile* src_file = (PyFile*)pysrc_file; if (src_file->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = tkrzw::ImportDBMFromFlatRecords(self->dbm, src_file->file); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#ExportKeysAsLines. static PyObject* dbm_ExportKeysAsLines(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydest_file = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydest_file, cls_file)) { ThrowInvalidArguments("the argument is not a File"); return nullptr; } PyFile* dest_file = (PyFile*)pydest_file; if (dest_file->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = tkrzw::ExportDBMKeysAsLines(self->dbm, dest_file->file); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#Inspect. static PyObject* dbm_Inspect(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } std::vector> records; { NativeLock lock(self->concurrent); records = self->dbm->Inspect(); } PyObject* pyrv = PyDict_New(); for (const auto& rec : records) { PyObject* pyname = CreatePyString(rec.first); PyObject* pyvalue = CreatePyString(rec.second); PyDict_SetItem(pyrv, pyname, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pyname); } return pyrv; } // Implementation of DBM#IsOpen. static PyObject* dbm_IsOpen(PyDBM* self) { if (self->dbm == nullptr) { Py_RETURN_FALSE; } Py_RETURN_TRUE; } // Implementation of DBM#IsWritable. static PyObject* dbm_IsWritable(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const bool writable = self->dbm->IsWritable(); if (writable) { Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of DBM#IsHealthy. static PyObject* dbm_IsHealthy(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const bool healthy = self->dbm->IsHealthy(); if (healthy) { Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of DBM#IsOrdered. static PyObject* dbm_IsOrdered(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const bool ordered = self->dbm->IsOrdered(); if (ordered) { Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of DBM#Search. static PyObject* dbm_Search(PyDBM* self, PyObject* pyargs) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pymode = PyTuple_GET_ITEM(pyargs, 0); PyObject* pypattern = PyTuple_GET_ITEM(pyargs, 1); int32_t capacity = 0; if (argc > 2) { capacity = PyObjToInt(PyTuple_GET_ITEM(pyargs, 2)); } SoftString pattern(pypattern); SoftString mode(pymode); std::vector keys; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = tkrzw::SearchDBMModal(self->dbm, mode.Get(), pattern.Get(), &keys, capacity); } if (status != tkrzw::Status::SUCCESS) { ThrowStatusException(status); return nullptr; } PyObject* pyrv = PyList_New(keys.size()); for (size_t i = 0; i < keys.size(); i++) { PyList_SET_ITEM(pyrv, i, CreatePyString(keys[i])); } return pyrv; } // Implementation of DBM#MakeIterator. static PyObject* dbm_MakeIterator(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } PyTypeObject* pyitertype = (PyTypeObject*)cls_iter; PyIterator* pyiter = (PyIterator*)pyitertype->tp_alloc(pyitertype, 0); if (!pyiter) return nullptr; { NativeLock lock(self->concurrent); pyiter->iter = self->dbm->MakeIterator().release(); } pyiter->concurrent = self->concurrent; return (PyObject*)pyiter; } // Implementation of DBM.RestoreDatabase. static PyObject* dbm_RestoreDatabase(PyObject* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 5) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } SoftString old_file_path(PyTuple_GET_ITEM(pyargs, 0)); SoftString new_file_path(PyTuple_GET_ITEM(pyargs, 1)); SoftString class_name(argc > 2 ? PyTuple_GET_ITEM(pyargs, 2) : Py_None); const int64_t end_offset = argc > 3 ? PyObjToInt(PyTuple_GET_ITEM(pyargs, 3)) : -1; SoftString cipher_key(argc > 4 ? PyTuple_GET_ITEM(pyargs, 4) : Py_None); tkrzw::Status status(tkrzw::Status::SUCCESS); int32_t num_shards = 0; if (tkrzw::ShardDBM::GetNumberOfShards(std::string(old_file_path.Get()), &num_shards) == tkrzw::Status::SUCCESS) { NativeLock lock(true); status = tkrzw::ShardDBM::RestoreDatabase( std::string(old_file_path.Get()), std::string(new_file_path.Get()), std::string(class_name.Get()), end_offset, cipher_key.Get()); } else { NativeLock lock(true); status = tkrzw::PolyDBM::RestoreDatabase( std::string(old_file_path.Get()), std::string(new_file_path.Get()), std::string(class_name.Get()), end_offset, cipher_key.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of DBM#__len__. static Py_ssize_t dbm_len(PyDBM* self) { if (self->dbm == nullptr) { return 0; } int64_t count = -1; { NativeLock lock(self->concurrent); count = self->dbm->CountSimple(); } return std::max(count, 0); } // Implementation of DBM#__getitem__. static PyObject* dbm_getitem(PyDBM* self, PyObject* pykey) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } const bool is_unicode = PyUnicode_Check(pykey); SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); std::string value; { NativeLock lock(self->concurrent); status = self->dbm->Get(key.Get(), &value); } if (status != tkrzw::Status::SUCCESS) { ThrowStatusException(status); return nullptr; } if (is_unicode) { return CreatePyString(value); } return CreatePyBytes(value); } // Implementation of DBM#__contains__. static int dbm_contains(PyDBM* self, PyObject* pykey) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return -1; } SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Get(key.Get()); } if (status == tkrzw::Status::SUCCESS) { return 1; } if (status == tkrzw::Status::NOT_FOUND_ERROR) { return 0; } return -1; } // Implementation of DBM#__setitem__ and DBM#__delitem__. static int dbm_setitem(PyDBM* self, PyObject* pykey, PyObject* pyvalue) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return -1; } if (pyvalue) { SoftString key(pykey); SoftString value(pyvalue); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Set(key.Get(), value.Get()); } if (status != tkrzw::Status::SUCCESS) { ThrowStatusException(status); return -1; } } else { SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->dbm->Remove(key.Get()); } if (status != tkrzw::Status::SUCCESS) { ThrowStatusException(status); return -1; } } return 0; } // Implementation of DBM#__iter__. static PyObject* dbm_iter(PyDBM* self) { if (self->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } PyTypeObject* pyitertype = (PyTypeObject*)cls_iter; PyIterator* pyiter = (PyIterator*)pyitertype->tp_alloc(pyitertype, 0); if (!pyiter) return nullptr; { NativeLock lock(self->concurrent); pyiter->iter = self->dbm->MakeIterator().release(); pyiter->concurrent = self->concurrent; pyiter->iter->First(); } return (PyObject*)pyiter; } // Defines the DBM class. static bool DefineDBM() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.DBM"; pytype.tp_basicsize = sizeof(PyDBM); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Polymorphic database manager."; pytype.tp_new = dbm_new; pytype.tp_dealloc = (destructor)dbm_dealloc; pytype.tp_init = (initproc)dbm_init; pytype.tp_repr = (unaryfunc)dbm_repr; pytype.tp_str = (unaryfunc)dbm_str; static PyMethodDef methods[] = { {"Open", (PyCFunction)dbm_Open, METH_VARARGS | METH_KEYWORDS, "Opens a database file."}, {"Close", (PyCFunction)dbm_Close, METH_NOARGS, "Closes the database file."}, {"Process", (PyCFunction)dbm_Process, METH_VARARGS, "Processes a record with an arbitrary function."}, {"Get", (PyCFunction)dbm_Get, METH_VARARGS, "Gets the value of a record of a key."}, {"GetStr", (PyCFunction)dbm_GetStr, METH_VARARGS, "Gets the value of a record of a key, as a string."}, {"GetMulti", (PyCFunction)dbm_GetMulti, METH_VARARGS, "Gets the values of multiple records of keys."}, {"GetMultiStr", (PyCFunction)dbm_GetMultiStr, METH_VARARGS, "Gets the values of multiple records of keys, as strings."}, {"Set", (PyCFunction)dbm_Set, METH_VARARGS, "Sets a record of a key and a value."}, {"SetMulti", (PyCFunction)dbm_SetMulti, METH_VARARGS | METH_KEYWORDS, "Sets multiple records specified by an initializer list of pairs of strings."}, {"SetAndGet", (PyCFunction)dbm_SetAndGet, METH_VARARGS, "Sets a record and get the old value."}, {"Remove", (PyCFunction)dbm_Remove, METH_VARARGS, "Removes a record of a key."}, {"RemoveMulti", (PyCFunction)dbm_RemoveMulti, METH_VARARGS, "Removes records of keys."}, {"RemoveAndGet", (PyCFunction)dbm_RemoveAndGet, METH_VARARGS, "Removes a record and get the value."}, {"Append", (PyCFunction)dbm_Append, METH_VARARGS, "Appends data at the end of a record of a key."}, {"AppendMulti", (PyCFunction)dbm_AppendMulti, METH_VARARGS | METH_KEYWORDS, "Appends data to multiple records of the keyword arguments."}, {"CompareExchange", (PyCFunction)dbm_CompareExchange, METH_VARARGS, "Compares the value of a record and exchanges if the condition meets."}, {"CompareExchangeAndGet", (PyCFunction)dbm_CompareExchangeAndGet, METH_VARARGS, "Does compare-and-exchange and/or gets the old value of the record."}, {"Increment", (PyCFunction)dbm_Increment, METH_VARARGS, "Increments the numeric value of a record."}, {"ProcessMulti", (PyCFunction)dbm_ProcessMulti, METH_VARARGS, "Processes multiple records with arbitrary functions."}, {"CompareExchangeMulti", (PyCFunction)dbm_CompareExchangeMulti, METH_VARARGS, "Compares the values of records and exchanges if the condition meets."}, {"Rekey", (PyCFunction)dbm_Rekey, METH_VARARGS, "Changes the key of a record."}, {"PopFirst", (PyCFunction)dbm_PopFirst, METH_VARARGS, "Gets the first record and removes it."}, {"PopFirstStr", (PyCFunction)dbm_PopFirstStr, METH_VARARGS, "Gets the first record as strings and removes it."}, {"PushLast", (PyCFunction)dbm_PushLast, METH_VARARGS, "Adds a record with a key of the current timestamp."}, {"ProcessEach", (PyCFunction)dbm_ProcessEach, METH_VARARGS, "Processes each and every record in the database with an arbitrary function."}, {"Count", (PyCFunction)dbm_Count, METH_NOARGS, "Gets the number of records."}, {"GetFileSize", (PyCFunction)dbm_GetFileSize, METH_NOARGS, "Gets the current file size of the database."}, {"GetFilePath", (PyCFunction)dbm_GetFilePath, METH_NOARGS, "Gets the path of the database file."}, {"GetTimestamp", (PyCFunction)dbm_GetTimestamp, METH_NOARGS, "Gets the timestamp in seconds of the last modified time."}, {"Clear", (PyCFunction)dbm_Clear, METH_NOARGS, "Removes all records."}, {"Rebuild", (PyCFunction)dbm_Rebuild, METH_VARARGS | METH_KEYWORDS, "Rebuilds the entire database."}, {"ShouldBeRebuilt", (PyCFunction)dbm_ShouldBeRebuilt, METH_NOARGS, "Checks whether the database should be rebuilt."}, {"Synchronize", (PyCFunction)dbm_Synchronize, METH_VARARGS | METH_KEYWORDS, "Synchronizes the content of the database to the file system."}, {"CopyFileData", (PyCFunction)dbm_CopyFileData, METH_VARARGS, "Copies the content of the database file to another file."}, {"Export", (PyCFunction)dbm_Export, METH_VARARGS, "Exports all records to another database."}, {"ExportToFlatRecords", (PyCFunction)dbm_ExportToFlatRecords, METH_VARARGS, "Exports all records of a database to a flat record file."}, {"ImportFromFlatRecords", (PyCFunction)dbm_ImportFromFlatRecords, METH_VARARGS, "Imports records to a database from a flat record file."}, {"ExportKeysAsLines", (PyCFunction)dbm_ExportKeysAsLines, METH_VARARGS, "Exports the keys of all records as lines to a text file."}, {"Inspect", (PyCFunction)dbm_Inspect, METH_NOARGS, "Inspects the database."}, {"IsOpen", (PyCFunction)dbm_IsOpen, METH_NOARGS, "Checks whether the database is open."}, {"IsWritable", (PyCFunction)dbm_IsWritable, METH_NOARGS, "Checks whether the database is writable."}, {"IsHealthy", (PyCFunction)dbm_IsHealthy, METH_NOARGS, "Checks whether the database condition is healthy."}, {"IsOrdered", (PyCFunction)dbm_IsOrdered, METH_NOARGS, "Checks whether ordered operations are supported."}, {"Search", (PyCFunction)dbm_Search, METH_VARARGS, "Searches the database and get keys which match a pattern."}, {"MakeIterator", (PyCFunction)dbm_MakeIterator, METH_NOARGS, "Makes an iterator for each record."}, {"RestoreDatabase", (PyCFunction)dbm_RestoreDatabase, METH_CLASS | METH_VARARGS, "Makes an iterator for each record."}, {nullptr, nullptr, 0, nullptr}, }; pytype.tp_methods = methods; static PyMappingMethods map_methods; std::memset(&map_methods, 0, sizeof(map_methods)); map_methods.mp_length = (lenfunc)dbm_len; map_methods.mp_subscript = (binaryfunc)dbm_getitem; map_methods.mp_ass_subscript = (objobjargproc)dbm_setitem; pytype.tp_as_mapping = &map_methods; static PySequenceMethods seq_methods; std::memset(&seq_methods, 0, sizeof(seq_methods)); seq_methods.sq_contains = (objobjproc)dbm_contains; pytype.tp_as_sequence = &seq_methods; pytype.tp_iter = (getiterfunc)dbm_iter; if (PyType_Ready(&pytype) != 0) return false; cls_dbm = (PyObject*)&pytype; Py_INCREF(cls_dbm); obj_dbm_any_data = PyBytes_FromStringAndSize("\0[ANY]\0", 7); if (PyObject_GenericSetAttr( cls_dbm, PyUnicode_FromString("ANY_DATA"), obj_dbm_any_data) != 0) { return false; } if (PyModule_AddObject(mod_tkrzw, "DBM", cls_dbm) != 0) return false; return true; } // Implementation of Iterator.new. static PyObject* iter_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyIterator* self = (PyIterator*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->iter = nullptr; self->concurrent = false; return (PyObject*)self; } // Implementation of Iterator#dealloc. static void iter_dealloc(PyIterator* self) { delete self->iter; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of Iterator#__init__. static int iter_init(PyIterator* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return -1; } PyObject* pydbm_obj = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydbm_obj, cls_dbm)) { ThrowInvalidArguments("the argument is not a DBM"); return -1; } PyDBM* pydbm = (PyDBM*)pydbm_obj; { NativeLock lock(pydbm->concurrent); self->iter = pydbm->dbm->MakeIterator().release(); } self->concurrent = pydbm->concurrent; return 0; } // Implementation of Iterator#__repr__. static PyObject* iter_repr(PyIterator* self) { std::string key; { NativeLock lock(self->concurrent); const tkrzw::Status status = self->iter->Get(&key); if (status != tkrzw::Status::SUCCESS) { key = "(unlocated)"; } } return CreatePyString(tkrzw::StrCat( "")); } // Implementation of Iterator#__str__. static PyObject* iter_str(PyIterator* self) { std::string key; { NativeLock lock(self->concurrent); const tkrzw::Status status = self->iter->Get(&key); if (status != tkrzw::Status::SUCCESS) { key = "(unlocated)"; } } return CreatePyString(tkrzw::StrEscapeC(key, true)); } // Implementation of Iterator#First. static PyObject* iter_First(PyIterator* self) { tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->First(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Last. static PyObject* iter_Last(PyIterator* self) { tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Last(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Jump. static PyObject* iter_Jump(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Jump(key.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#JumpLower. static PyObject* iter_JumpLower(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); const bool inclusive = argc > 1 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 1)) : false; SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->JumpLower(key.Get(), inclusive); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#JumpUpper. static PyObject* iter_JumpUpper(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); const bool inclusive = argc > 1 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 1)) : false; SoftString key(pykey); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->JumpUpper(key.Get(), inclusive); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Next. static PyObject* iter_Next(PyIterator* self) { tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Next(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Previous. static PyObject* iter_Previous(PyIterator* self) { tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Previous(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Get. static PyObject* iter_Get(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(&key, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyBytes(key); PyObject* pyvalue = CreatePyBytes(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of Iterator#GetStr. static PyObject* iter_GetStr(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(&key, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyString(key); PyObject* pyvalue = CreatePyString(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of Iterator#GetKey. static PyObject* iter_GetKey(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(&key); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return CreatePyBytes(key); } // Implementation of Iterator#GetKeyStr. static PyObject* iter_GetKeyStr(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(&key); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return CreatePyString(key); } // Implementation of Iterator#GetValue. static PyObject* iter_GetValue(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(nullptr, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return CreatePyBytes(value); } // Implementation of Iterator#GetValueStr. static PyObject* iter_GetValueStr(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(nullptr, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return CreatePyString(value); } // Implementation of Iterator#Set. static PyObject* iter_Set(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 0); SoftString value(pyvalue); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Set(value.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Remove. static PyObject* iter_Remove(PyIterator* self) { tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Remove(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Iterator#Step. static PyObject* iter_Step(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Step(&key, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyBytes(key); PyObject* pyvalue = CreatePyBytes(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of Iterator#StepStr. static PyObject* iter_StepStr(PyIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pystatus = nullptr; if (argc > 0) { pystatus = PyTuple_GET_ITEM(pyargs, 0); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Step(&key, &value); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyString(key); PyObject* pyvalue = CreatePyString(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of Iterator#__next__. static PyObject* iter_iternext(PyIterator* self) { std::string key, value; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->iter->Get(&key, &value); } PyObject* pyrv = nullptr; if (status == tkrzw::Status::SUCCESS) { PyObject* pykey = CreatePyBytes(key); PyObject* pyvalue = CreatePyBytes(value); pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pykey); Py_DECREF(pyvalue); self->iter->Next(); } else { PyErr_SetString(PyExc_StopIteration, "end of iteration"); pyrv = nullptr; } return pyrv; } // Defines the Iterator class. static bool DefineIterator() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.Iterator"; pytype.tp_basicsize = sizeof(PyIterator); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Iterator for each record."; pytype.tp_new = iter_new; pytype.tp_dealloc = (destructor)iter_dealloc; pytype.tp_init = (initproc)iter_init; pytype.tp_repr = (unaryfunc)iter_repr; pytype.tp_str = (unaryfunc)iter_str; static PyMethodDef methods[] = { {"First", (PyCFunction)iter_First, METH_NOARGS, "Initializes the iterator to indicate the first record."}, {"Last", (PyCFunction)iter_Last, METH_NOARGS, "Initializes the iterator to indicate the last record."}, {"Jump", (PyCFunction)iter_Jump, METH_VARARGS, "Initializes the iterator to indicate a specific record."}, {"JumpLower", (PyCFunction)iter_JumpLower, METH_VARARGS, "Initializes the iterator to indicate the last record whose key is lower."}, {"JumpUpper", (PyCFunction)iter_JumpUpper, METH_VARARGS, "Initializes the iterator to indicate the first record whose key is upper."}, {"Next", (PyCFunction)iter_Next, METH_NOARGS, "Moves the iterator to the next record."}, {"Previous", (PyCFunction)iter_Previous, METH_NOARGS, "Moves the iterator to the previous record."}, {"Get", (PyCFunction)iter_Get, METH_VARARGS, "Gets the key and the value of the current record of the iterator."}, {"GetStr", (PyCFunction)iter_GetStr, METH_VARARGS, "Gets the key and the value of the current record of the iterator, as strings."}, {"GetKey", (PyCFunction)iter_GetKey, METH_VARARGS, "Gets the key of the current record."}, {"GetKeyStr", (PyCFunction)iter_GetKeyStr, METH_VARARGS, "Gets the key of the current record, as a string."}, {"GetValue", (PyCFunction)iter_GetValue, METH_VARARGS, "Gets the value of the current record."}, {"GetValueStr", (PyCFunction)iter_GetValueStr, METH_VARARGS, "Gets the value of the current record, as a string."}, {"Set", (PyCFunction)iter_Set, METH_VARARGS, "Sets the value of the current record."}, {"Remove", (PyCFunction)iter_Remove, METH_NOARGS, "Removes the current record."}, {"Step", (PyCFunction)iter_Step, METH_VARARGS, "Gets the current record and moves the iterator to the next record."}, {"StepStr", (PyCFunction)iter_StepStr, METH_VARARGS, "Gets the current record and moves the iterator to the next record, as strings."}, {nullptr, nullptr, 0, nullptr} }; pytype.tp_methods = methods; pytype.tp_iternext = (iternextfunc)iter_iternext; if (PyType_Ready(&pytype) != 0) return false; cls_iter = (PyObject*)&pytype; Py_INCREF(cls_iter); if (PyModule_AddObject(mod_tkrzw, "Iterator", cls_iter) != 0) return false; return true; } // Implementation of AsyncDBM.new. static PyObject* asyncdbm_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyAsyncDBM* self = (PyAsyncDBM*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->async = nullptr; self->concurrent = false; return (PyObject*)self; } // Implementation of AsyncDBM#dealloc. static void asyncdbm_dealloc(PyAsyncDBM* self) { delete self->async; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of AsyncDBM#__init__. static int asyncdbm_init(PyAsyncDBM* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return -1; } PyObject* pydbm = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydbm, cls_dbm)) { ThrowInvalidArguments("the argument is not a DBM"); return -1; } PyDBM* dbm = (PyDBM*)pydbm; if (dbm->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return -1; } PyObject* pynum_threads = PyTuple_GET_ITEM(pyargs, 1); const int32_t num_threads = PyObjToInt(pynum_threads); self->async = new tkrzw::AsyncDBM(dbm->dbm, num_threads); self->concurrent = dbm->concurrent; return 0; } // Implementation of AsyncDBM#__repr__. static PyObject* asyncdbm_repr(PyAsyncDBM* self) { const std::string& str = tkrzw::SPrintF("", (void*)self->async); return CreatePyString(str); } // Implementation of AsyncDBM#__str__. static PyObject* asyncdbm_str(PyAsyncDBM* self) { const std::string& str = tkrzw::SPrintF("AsyncDBM:%p", (void*)self->async); return CreatePyString(str); } // Implementation of AsyncDBM#Destruct. static PyObject* asyncdbm_Destruct(PyAsyncDBM* self) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } delete self->async; self->async = nullptr; Py_RETURN_NONE; } // Implementation of AsyncDBM#Get. static PyObject* asyncdbm_Get(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); tkrzw::StatusFuture future(self->async->Get(key.Get())); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#GetStr. static PyObject* asyncdbm_GetStr(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); tkrzw::StatusFuture future(self->async->Get(key.Get())); return CreatePyFutureMove(std::move(future), self->concurrent, true); } // Implementation of AsyncDBM#GetMulti. static PyObject* asyncdbm_GetMulti(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); std::vector keys; for (int32_t i = 0; i < argc; i++) { PyObject* pykey = PyTuple_GET_ITEM(pyargs, i); SoftString key(pykey); keys.emplace_back(std::string(key.Get())); } std::vector key_views(keys.begin(), keys.end()); tkrzw::StatusFuture future(self->async->GetMulti(keys)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#GetMultiStr. static PyObject* asyncdbm_GetMultiStr(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); std::vector keys; for (int32_t i = 0; i < argc; i++) { PyObject* pykey = PyTuple_GET_ITEM(pyargs, i); SoftString key(pykey); keys.emplace_back(std::string(key.Get())); } std::vector key_views(keys.begin(), keys.end()); tkrzw::StatusFuture future(self->async->GetMulti(keys)); return CreatePyFutureMove(std::move(future), self->concurrent, true); } // Implementation of AsyncDBM#Set. static PyObject* asyncdbm_Set(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); const bool overwrite = argc > 2 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 2)) : true; SoftString key(pykey); SoftString value(pyvalue); tkrzw::StatusFuture future(self->async->Set(key.Get(), value.Get(), overwrite)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#SetMulti. static PyObject* asyncdbm_SetMulti(PyAsyncDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments("too many arguments"); return nullptr; } PyObject* pyoverwrite = argc > 0 ? PyTuple_GET_ITEM(pyargs, 0) : Py_True; const bool overwrite = PyObject_IsTrue(pyoverwrite); std::map records; if (pykwds != nullptr) { records = MapKeywords(pykwds); } std::map record_views; for (const auto& record : records) { record_views.emplace(std::make_pair( std::string_view(record.first), std::string_view(record.second))); } tkrzw::StatusFuture future(self->async->SetMulti(record_views, overwrite)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Remove. static PyObject* asyncdbm_Remove(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); tkrzw::StatusFuture future(self->async->Remove(key.Get())); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#RemoveMulti. static PyObject* asyncdbm_RemoveMulti(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); std::vector keys; for (int32_t i = 0; i < argc; i++) { PyObject* pykey = PyTuple_GET_ITEM(pyargs, i); SoftString key(pykey); keys.emplace_back(std::string(key.Get())); } std::vector key_views(keys.begin(), keys.end()); tkrzw::StatusFuture future(self->async->RemoveMulti(key_views)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Append. static PyObject* asyncdbm_Append(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); PyObject* pydelim = argc > 2 ? PyTuple_GET_ITEM(pyargs, 2) : nullptr; SoftString key(pykey); SoftString value(pyvalue); SoftString delim(pydelim == nullptr ? Py_None : pydelim); tkrzw::StatusFuture future(self->async->Append(key.Get(), value.Get(), delim.Get())); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#AppendMulti. static PyObject* asyncdbm_AppendMulti(PyAsyncDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc > 1) { ThrowInvalidArguments("too many arguments"); return nullptr; } PyObject* pydelim = argc > 0 ? PyTuple_GET_ITEM(pyargs, 0) : nullptr; SoftString delim(pydelim == nullptr ? Py_None : pydelim); std::map records; if (pykwds != nullptr) { records = MapKeywords(pykwds); } std::map record_views; for (const auto& record : records) { record_views.emplace(std::make_pair( std::string_view(record.first), std::string_view(record.second))); } tkrzw::StatusFuture future(self->async->AppendMulti(record_views, delim.Get())); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#CompareExchange. static PyObject* asyncdbm_CompareExchange(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 3) { ThrowInvalidArguments(argc < 3 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyexpected = PyTuple_GET_ITEM(pyargs, 1); PyObject* pydesired = PyTuple_GET_ITEM(pyargs, 2); SoftString key(pykey); std::unique_ptr expected; std::string_view expected_view; if (pyexpected != Py_None) { if (pyexpected == obj_dbm_any_data) { expected_view = tkrzw::DBM::ANY_DATA; } else { expected = std::make_unique(pyexpected); expected_view = expected->Get(); } } std::unique_ptr desired; std::string_view desired_view; if (pydesired != Py_None) { if (pydesired == obj_dbm_any_data) { desired_view = tkrzw::DBM::ANY_DATA; } else { desired = std::make_unique(pydesired); desired_view = desired->Get(); } } tkrzw::StatusFuture future(self->async->CompareExchange( key.Get(), expected_view, desired_view)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Increment. static PyObject* asyncdbm_Increment(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 3) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); int64_t inc = 1; if (argc > 1) { PyObject* pyinc = PyTuple_GET_ITEM(pyargs, 1); inc = PyObjToInt(pyinc); } int64_t init = 0; if (argc > 2) { PyObject* pyinit = PyTuple_GET_ITEM(pyargs, 2); init = PyObjToInt(pyinit); } tkrzw::StatusFuture future(self->async->Increment(key.Get(), inc, init)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#CompareExchangeMulti. static PyObject* asyncdbm_CompareExchangeMulti(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyexpected = PyTuple_GET_ITEM(pyargs, 0); PyObject* pydesired = PyTuple_GET_ITEM(pyargs, 1); if (!PySequence_Check(pyexpected) || !PySequence_Check(pydesired)) { ThrowInvalidArguments("parameters must be sequences of strings"); return nullptr; } std::vector expected_ph; const auto& expected = ExtractSVPairs(pyexpected, &expected_ph); std::vector desired_ph; const auto& desired = ExtractSVPairs(pydesired, &desired_ph); tkrzw::StatusFuture future(self->async->CompareExchangeMulti(expected, desired)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Rekey. static PyObject* asyncdbm_Rekey(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 4) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyold_key = PyTuple_GET_ITEM(pyargs, 0); PyObject* pynew_key = PyTuple_GET_ITEM(pyargs, 1); const bool overwrite = argc > 2 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 2)) : true; const bool copying = argc > 3 ? PyObject_IsTrue(PyTuple_GET_ITEM(pyargs, 3)) : false; SoftString old_key(pyold_key); SoftString new_key(pynew_key); tkrzw::StatusFuture future(self->async->Rekey( old_key.Get(), new_key.Get(), overwrite, copying)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#PopFirst. static PyObject* asyncdbm_PopFirst(PyAsyncDBM* self) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } tkrzw::StatusFuture future(self->async->PopFirst()); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#PopFirstStr. static PyObject* asyncdbm_PopFirstStr(PyAsyncDBM* self) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } tkrzw::StatusFuture future(self->async->PopFirst()); return CreatePyFutureMove(std::move(future), self->concurrent, true); } // Implementation of AsyncDBM#PushLast. static PyObject* asyncdbm_PushLast(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 0); const double wtime = argc > 1 ? PyObjToDouble(PyTuple_GET_ITEM(pyargs, 1)) : -1; SoftString value(pyvalue); tkrzw::StatusFuture future(self->async->PushLast(value.Get(), wtime)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Clear. static PyObject* asyncdbm_Clear(PyAsyncDBM* self) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } tkrzw::StatusFuture future(self->async->Clear()); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Rebuild. static PyObject* asyncdbm_Rebuild(PyAsyncDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 0) { ThrowInvalidArguments("too many arguments"); return nullptr; } std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); } tkrzw::StatusFuture future(self->async->Rebuild(params)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Synchronize. static PyObject* asyncdbm_Synchronize(PyAsyncDBM* self, PyObject* pyargs, PyObject* pykwds) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyhard = PyTuple_GET_ITEM(pyargs, 0); const bool hard = PyObject_IsTrue(pyhard); std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); } tkrzw::StatusFuture future(self->async->Synchronize(hard, nullptr, params)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#CopyFileData. static PyObject* asyncdbm_CopyFileData(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } bool sync_hard = false; if (argc > 1) { PyObject* pysync_hard = PyTuple_GET_ITEM(pyargs, 1); sync_hard = PyObject_IsTrue(pysync_hard); } PyObject* pydest = PyTuple_GET_ITEM(pyargs, 0); SoftString dest(pydest); tkrzw::StatusFuture future(self->async->CopyFileData(std::string(dest.Get()), sync_hard)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Export. static PyObject* asyncdbm_Export(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydest = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydest, cls_dbm)) { ThrowInvalidArguments("the argument is not a DBM"); return nullptr; } PyDBM* dest = (PyDBM*)pydest; if (dest->dbm == nullptr) { ThrowInvalidArguments("not opened database"); return nullptr; } tkrzw::StatusFuture future(self->async->Export(dest->dbm)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#ExportToFlatRecords. static PyObject* asyncdbm_ExportToFlatRecords(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydest_file = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pydest_file, cls_file)) { ThrowInvalidArguments("the argument is not a File"); return nullptr; } PyFile* dest_file = (PyFile*)pydest_file; if (dest_file->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } tkrzw::StatusFuture future(self->async->ExportToFlatRecords(dest_file->file)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#ImportFromFlatRecords. static PyObject* asyncdbm_ImportFromFlatRecords(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pysrc_file = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pysrc_file, cls_file)) { ThrowInvalidArguments("the argument is not a File"); return nullptr; } PyFile* src_file = (PyFile*)pysrc_file; if (src_file->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } tkrzw::StatusFuture future(self->async->ImportFromFlatRecords(src_file->file)); return CreatePyFutureMove(std::move(future), self->concurrent); } // Implementation of AsyncDBM#Search. static PyObject* asyncdbm_Search(PyAsyncDBM* self, PyObject* pyargs) { if (self->async == nullptr) { ThrowInvalidArguments("destructed object"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pymode = PyTuple_GET_ITEM(pyargs, 0); PyObject* pypattern = PyTuple_GET_ITEM(pyargs, 1); int32_t capacity = 0; if (argc > 2) { capacity = PyObjToInt(PyTuple_GET_ITEM(pyargs, 2)); } SoftString pattern(pypattern); SoftString mode(pymode); tkrzw::StatusFuture future(self->async->SearchModal(mode.Get(), pattern.Get(), capacity)); return CreatePyFutureMove(std::move(future), self->concurrent, true); } // Defines the AsyncDBM class. static bool DefineAsyncDBM() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.AsyncDBM"; pytype.tp_basicsize = sizeof(PyAsyncDBM); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Polymorphic database manager."; pytype.tp_new = asyncdbm_new; pytype.tp_dealloc = (destructor)asyncdbm_dealloc; pytype.tp_init = (initproc)asyncdbm_init; pytype.tp_repr = (unaryfunc)asyncdbm_repr; pytype.tp_str = (unaryfunc)asyncdbm_str; static PyMethodDef methods[] = { {"Destruct", (PyCFunction)asyncdbm_Destruct, METH_NOARGS, "Destructs the asynchronous database adapter."}, {"Get", (PyCFunction)asyncdbm_Get, METH_VARARGS, "Gets the value of a record of a key."}, {"GetStr", (PyCFunction)asyncdbm_GetStr, METH_VARARGS, "Gets the value of a record of a key, as a string."}, {"GetMulti", (PyCFunction)asyncdbm_GetMulti, METH_VARARGS, "Gets the values of multiple records of keys."}, {"GetMultiStr", (PyCFunction)asyncdbm_GetMultiStr, METH_VARARGS, "Gets the values of multiple records of keys, as strings."}, {"Set", (PyCFunction)asyncdbm_Set, METH_VARARGS, "Sets a record of a key and a value."}, {"SetMulti", (PyCFunction)asyncdbm_SetMulti, METH_VARARGS | METH_KEYWORDS, "Sets multiple records specified by an initializer list of pairs of strings."}, {"Remove", (PyCFunction)asyncdbm_Remove, METH_VARARGS, "Removes a record of a key."}, {"RemoveMulti", (PyCFunction)asyncdbm_RemoveMulti, METH_VARARGS, "Removes records of keys."}, {"Append", (PyCFunction)asyncdbm_Append, METH_VARARGS, "Appends data at the end of a record of a key."}, {"AppendMulti", (PyCFunction)asyncdbm_AppendMulti, METH_VARARGS | METH_KEYWORDS, "Appends data to multiple records of the keyword arguments."}, {"CompareExchange", (PyCFunction)asyncdbm_CompareExchange, METH_VARARGS, "Compares the value of a record and exchanges if the condition meets."}, {"Increment", (PyCFunction)asyncdbm_Increment, METH_VARARGS, "Increments the numeric value of a record."}, {"CompareExchangeMulti", (PyCFunction)asyncdbm_CompareExchangeMulti, METH_VARARGS, "Compares the values of records and exchanges if the condition meets."}, {"Rekey", (PyCFunction)asyncdbm_Rekey, METH_VARARGS, "Changes the key of a record."}, {"PopFirst", (PyCFunction)asyncdbm_PopFirst, METH_NOARGS, "Gets the first record and removes it."}, {"PopFirstStr", (PyCFunction)asyncdbm_PopFirstStr, METH_NOARGS, "Gets the first record as strings and removes it."}, {"PushLast", (PyCFunction)asyncdbm_PushLast, METH_VARARGS, "Adds a record with a key of the current timestamp."}, {"Clear", (PyCFunction)asyncdbm_Clear, METH_NOARGS, "Removes all records."}, {"Rebuild", (PyCFunction)asyncdbm_Rebuild, METH_VARARGS | METH_KEYWORDS, "Rebuilds the entire database."}, {"Synchronize", (PyCFunction)asyncdbm_Synchronize, METH_VARARGS | METH_KEYWORDS, "Synchronizes the content of the database to the file system."}, {"CopyFileData", (PyCFunction)asyncdbm_CopyFileData, METH_VARARGS, "Copies the content of the database file to another file."}, {"Export", (PyCFunction)asyncdbm_Export, METH_VARARGS, "Exports all records to another database."}, {"ExportToFlatRecords", (PyCFunction)asyncdbm_ExportToFlatRecords, METH_VARARGS, "Exports all records of a database to a flat record file."}, {"ImportFromFlatRecords", (PyCFunction)asyncdbm_ImportFromFlatRecords, METH_VARARGS, "Imports records to a database from a flat record file."}, {"Search", (PyCFunction)asyncdbm_Search, METH_VARARGS, "Searches the database and get keys which match a pattern."}, {nullptr, nullptr, 0, nullptr}, }; pytype.tp_methods = methods; if (PyType_Ready(&pytype) != 0) return false; cls_asyncdbm = (PyObject*)&pytype; Py_INCREF(cls_asyncdbm); if (PyModule_AddObject(mod_tkrzw, "AsyncDBM", cls_asyncdbm) != 0) return false; return true; } // Implementation of File.new. static PyObject* file_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyFile* self = (PyFile*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->file = nullptr; self->concurrent = false; return (PyObject*)self; } // Implementation of File#dealloc. static void file_dealloc(PyFile* self) { delete self->file; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of File#__init__. static int file_init(PyFile* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 0) { ThrowInvalidArguments("too many arguments"); return -1; } return 0; } // Implementation of File#__repr__. static PyObject* file_repr(PyFile* self) { if (self->file == nullptr) { return CreatePyString(""); } std::string class_name = "unknown"; auto* in_file = self->file->GetInternalFile(); if (in_file != nullptr) { const auto& file_type = in_file->GetType(); if (file_type == typeid(tkrzw::StdFile)) { class_name = "StdFile"; } else if (file_type == typeid(tkrzw::MemoryMapParallelFile)) { class_name = "MemoryMapParallelFile"; } else if (file_type == typeid(tkrzw::MemoryMapAtomicFile)) { class_name = "MemoryMapAtomicFile"; } else if (file_type == typeid(tkrzw::PositionalParallelFile)) { class_name = "PositionalParallelFile"; } else if (file_type == typeid(tkrzw::PositionalAtomicFile)) { class_name = "PositionalAtomicFile"; } } const std::string path = self->file->GetPathSimple(); const int64_t size = self->file->GetSizeSimple(); const std::string& str = tkrzw::StrCat( ""); return CreatePyString(str); } // Implementation of File#__str__. static PyObject* file_str(PyFile* self) { if (self->file == nullptr) { return CreatePyString("(unopened)"); } std::string class_name = "unknown"; auto* in_file = self->file->GetInternalFile(); if (in_file != nullptr) { const auto& file_type = in_file->GetType(); if (file_type == typeid(tkrzw::StdFile)) { class_name = "StdFile"; } else if (file_type == typeid(tkrzw::MemoryMapParallelFile)) { class_name = "MemoryMapParallelFile"; } else if (file_type == typeid(tkrzw::MemoryMapAtomicFile)) { class_name = "MemoryMapAtomicFile"; } else if (file_type == typeid(tkrzw::PositionalParallelFile)) { class_name = "PositionalParallelFile"; } else if (file_type == typeid(tkrzw::PositionalAtomicFile)) { class_name = "PositionalAtomicFile"; } } const std::string path = self->file->GetPathSimple(); const int64_t size = self->file->GetSizeSimple(); const std::string& str = tkrzw::StrCat( "class=", class_name, " path=", tkrzw::StrEscapeC(path, true), " size=", size); return CreatePyString(str); } // Implementation of File#Open. static PyObject* file_Open(PyFile* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pypath = PyTuple_GET_ITEM(pyargs, 0); PyObject* pywritable = PyTuple_GET_ITEM(pyargs, 1); SoftString path(pypath); const bool writable = PyObject_IsTrue(pywritable); bool concurrent = false; int32_t open_options = 0; std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); if (tkrzw::StrToBool(tkrzw::SearchMap(params, "concurrent", "false"))) { concurrent = true; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "truncate", "false"))) { open_options |= tkrzw::File::OPEN_TRUNCATE; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_create", "false"))) { open_options |= tkrzw::File::OPEN_NO_CREATE; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_wait", "false"))) { open_options |= tkrzw::File::OPEN_NO_WAIT; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_lock", "false"))) { open_options |= tkrzw::File::OPEN_NO_LOCK; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "sync_hard", "false"))) { open_options |= tkrzw::File::OPEN_SYNC_HARD; } params.erase("concurrent"); params.erase("truncate"); params.erase("no_create"); params.erase("no_wait"); params.erase("no_lock"); params.erase("sync_hard"); } self->file = new tkrzw::PolyFile(); self->concurrent = concurrent; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->OpenAdvanced(std::string(path.Get()), writable, open_options, params); } if (status != tkrzw::Status::SUCCESS) { delete self->file; self->file = nullptr; } return CreatePyTkStatusMove(std::move(status)); } // Implementation of File#Close static PyObject* file_Close(PyFile* self) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Close(); } delete self->file; self->file = nullptr; return CreatePyTkStatusMove(std::move(status)); } static PyObject* file_Read(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } const int64_t off = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 0))); const int64_t size = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 1))); PyObject* pystatus = nullptr; if (argc > 2) { pystatus = PyTuple_GET_ITEM(pyargs, 2); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } char* buf = new char[size]; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Read(off, buf, size); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { delete[] buf; Py_RETURN_NONE; } PyObject* pydata = CreatePyBytes(std::string_view(buf, size)); delete[] buf; return pydata; } static PyObject* file_ReadStr(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } const int64_t off = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 0))); const int64_t size = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 1))); PyObject* pystatus = nullptr; if (argc > 2) { pystatus = PyTuple_GET_ITEM(pyargs, 2); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } char* buf = new char[size]; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Read(off, buf, size); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { delete[] buf; Py_RETURN_NONE; } PyObject* pystr = CreatePyString(std::string_view(buf, size)); delete[] buf; return pystr; } static PyObject* file_Write(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } const int64_t off = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 0))); PyObject* pydata = PyTuple_GET_ITEM(pyargs, 1); SoftString data(pydata); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Write(off, data.Get().data(), data.Get().size()); } return CreatePyTkStatusMove(std::move(status)); } static PyObject* file_Append(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pydata = PyTuple_GET_ITEM(pyargs, 0); SoftString data(pydata); PyObject* pystatus = nullptr; if (argc > 2) { pystatus = PyTuple_GET_ITEM(pyargs, 1); if (pystatus == Py_None) { pystatus = nullptr; } else if (!PyObject_IsInstance(pystatus, cls_status)) { ThrowInvalidArguments("not a status object"); return nullptr; } } int64_t new_off = 0; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Append(data.Get().data(), data.Get().size(), &new_off); } if (pystatus != nullptr) { *((PyTkStatus*)pystatus)->status = status; } if (status != tkrzw::Status::SUCCESS) { Py_RETURN_NONE; } return PyLong_FromLongLong(new_off); } static PyObject* file_Truncate(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } const int64_t size = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 0))); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Truncate(size); } return CreatePyTkStatusMove(std::move(status)); } static PyObject* file_Synchronize(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 3) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyhard = PyTuple_GET_ITEM(pyargs, 0); const bool hard = PyObject_IsTrue(pyhard); int64_t off = 0; int64_t size = 0; if (argc > 1) { off = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 1))); } if (argc > 2) { size = std::max(0, PyObjToInt(PyTuple_GET_ITEM(pyargs, 2))); } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->Synchronize(hard, off, size); } return CreatePyTkStatusMove(std::move(status)); } static PyObject* file_GetSize(PyFile* self) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } int64_t size = -1; { NativeLock lock(self->concurrent); size = self->file->GetSizeSimple(); } if (size >= 0) { return PyLong_FromLongLong(size); } Py_RETURN_NONE; } static PyObject* file_GetPath(PyFile* self) { std::string path; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->file->GetPath(&path); } if (status == tkrzw::Status::SUCCESS) { return CreatePyString(path); } Py_RETURN_NONE; } // Implementation of File#Search. static PyObject* file_Search(PyFile* self, PyObject* pyargs) { if (self->file == nullptr) { ThrowInvalidArguments("not opened file"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 2 || argc > 3) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pymode = PyTuple_GET_ITEM(pyargs, 0); PyObject* pypattern = PyTuple_GET_ITEM(pyargs, 1); int32_t capacity = 0; if (argc > 2) { capacity = PyObjToInt(PyTuple_GET_ITEM(pyargs, 2)); } SoftString pattern(pypattern); SoftString mode(pymode); std::vector lines; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = tkrzw::SearchTextFileModal(self->file, mode.Get(), pattern.Get(), &lines, capacity); } if (status != tkrzw::Status::SUCCESS) { ThrowStatusException(status); return nullptr; } PyObject* pyrv = PyList_New(lines.size()); for (size_t i = 0; i < lines.size(); i++) { PyList_SET_ITEM(pyrv, i, CreatePyString(lines[i])); } return pyrv; } // Defines the File class. static bool DefineFile() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.File"; pytype.tp_basicsize = sizeof(PyFile); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Generic file implemenation."; pytype.tp_new = file_new; pytype.tp_dealloc = (destructor)file_dealloc; pytype.tp_init = (initproc)file_init; pytype.tp_repr = (unaryfunc)file_repr; pytype.tp_str = (unaryfunc)file_str; static PyMethodDef methods[] = { {"Open", (PyCFunction)file_Open, METH_VARARGS | METH_KEYWORDS, "Opens a text file."}, {"Close", (PyCFunction)file_Close, METH_NOARGS, "Closes the text file."}, {"Read", (PyCFunction)file_Read, METH_VARARGS, "Reads data."}, {"ReadStr", (PyCFunction)file_ReadStr, METH_VARARGS, "Reads data as a string."}, {"Write", (PyCFunction)file_Write, METH_VARARGS, "Writes data."}, {"Append", (PyCFunction)file_Append, METH_VARARGS, "Appends data at the end of the file."}, {"Truncate", (PyCFunction)file_Truncate, METH_VARARGS, "Truncates the file."}, {"Synchronize", (PyCFunction)file_Synchronize, METH_VARARGS, "Synchronizes the content of the file to the file system."}, {"GetSize", (PyCFunction)file_GetSize, METH_NOARGS, "Gets the size of the file."}, {"GetPath", (PyCFunction)file_GetPath, METH_NOARGS, "Gets the path of the file."}, {"Search", (PyCFunction)file_Search, METH_VARARGS, "Searches the text file and get lines which match a pattern."}, {nullptr, nullptr, 0, nullptr} }; pytype.tp_methods = methods; if (PyType_Ready(&pytype) != 0) return false; cls_file = (PyObject*)&pytype; Py_INCREF(cls_file); if (PyModule_AddObject(mod_tkrzw, "File", cls_file) != 0) return false; return true; } // Implementation of Index.new. static PyObject* index_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyIndex* self = (PyIndex*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->index = nullptr; self->concurrent = false; return (PyObject*)self; } // Implementation of Index#dealloc. static void index_dealloc(PyIndex* self) { delete self->index; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of Index#__init__. static int index_init(PyIndex* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 0) { ThrowInvalidArguments("too many arguments"); return -1; } return 0; } // Implementation of Index#__repr__. static PyObject* index_repr(PyIndex* self) { std::string path = "-"; int64_t num_records = -1; if (self->index != nullptr) { NativeLock lock(self->concurrent); path = self->index->GetFilePath(); num_records = self->index->Count(); } const std::string& str = tkrzw::StrCat( ""); return CreatePyString(str); } // Implementation of Index#__str__. static PyObject* index_str(PyIndex* self) { std::string path = "-"; int64_t num_records = -1; if (self->index != nullptr) { NativeLock lock(self->concurrent); path = self->index->GetFilePath(); num_records = self->index->Count(); } const std::string& str = tkrzw::StrCat( "path=", tkrzw::StrEscapeC(path, true), " num_records=", num_records); return CreatePyString(str); } // Implementation of Index#Open. static PyObject* index_Open(PyIndex* self, PyObject* pyargs, PyObject* pykwds) { if (self->index != nullptr) { ThrowInvalidArguments("opened index"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pypath = PyTuple_GET_ITEM(pyargs, 0); PyObject* pywritable = PyTuple_GET_ITEM(pyargs, 1); SoftString path(pypath); const bool writable = PyObject_IsTrue(pywritable); bool concurrent = false; int32_t open_options = 0; std::map params; if (pykwds != nullptr) { params = MapKeywords(pykwds); if (tkrzw::StrToBool(tkrzw::SearchMap(params, "concurrent", "false"))) { concurrent = true; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "truncate", "false"))) { open_options |= tkrzw::File::OPEN_TRUNCATE; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_create", "false"))) { open_options |= tkrzw::File::OPEN_NO_CREATE; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_wait", "false"))) { open_options |= tkrzw::File::OPEN_NO_WAIT; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "no_lock", "false"))) { open_options |= tkrzw::File::OPEN_NO_LOCK; } if (tkrzw::StrToBool(tkrzw::SearchMap(params, "sync_hard", "false"))) { open_options |= tkrzw::File::OPEN_SYNC_HARD; } params.erase("concurrent"); params.erase("truncate"); params.erase("no_create"); params.erase("no_wait"); params.erase("no_lock"); params.erase("sync_hard"); } self->index = new tkrzw::PolyIndex(); self->concurrent = concurrent; tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Open(std::string(path.Get()), writable, open_options, params); } if (status != tkrzw::Status::SUCCESS) { delete self->index; self->index = nullptr; } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#Close. static PyObject* index_Close(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Close(); } delete self->index; self->index = nullptr; return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#GetValues. static PyObject* index_GetValues(PyIndex* self, PyObject* pyargs) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); int32_t capacity = 0; if (argc > 1) { capacity = PyObjToInt(PyTuple_GET_ITEM(pyargs, 1)); } std::vector values; { NativeLock lock(self->concurrent); values = self->index->GetValues(key.Get(), capacity); } PyObject* pyrv = PyList_New(values.size()); for (size_t i = 0; i < values.size(); i++) { PyList_SET_ITEM(pyrv, i, CreatePyBytes(values[i])); } return pyrv; } // Implementation of Index#GetValuesStr. static PyObject* index_GetValuesStr(PyIndex* self, PyObject* pyargs) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); int32_t capacity = 0; if (argc > 1) { capacity = PyObjToInt(PyTuple_GET_ITEM(pyargs, 1)); } std::vector values; { NativeLock lock(self->concurrent); values = self->index->GetValues(key.Get(), capacity); } PyObject* pyrv = PyList_New(values.size()); for (size_t i = 0; i < values.size(); i++) { PyList_SET_ITEM(pyrv, i, CreatePyString(values[i])); } return pyrv; } // Implementation of Index#Add. static PyObject* index_Add(PyIndex* self, PyObject* pyargs) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); SoftString key(pykey); SoftString value(pyvalue); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Add(key.Get(), value.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#Remove. static PyObject* index_Remove(PyIndex* self, PyObject* pyargs) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 2) { ThrowInvalidArguments(argc < 2 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); SoftString key(pykey); SoftString value(pyvalue); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Remove(key.Get(), value.Get()); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#Count. static PyObject* index_Count(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } int64_t count = -1; { NativeLock lock(self->concurrent); count = self->index->Count(); } return PyLong_FromLongLong(count); } // Implementation of Index#GetFilePath. static PyObject* index_GetFilePath(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } std::string path; { NativeLock lock(self->concurrent); path = self->index->GetFilePath(); } return CreatePyString(path); } // Implementation of Index#Clear. static PyObject* index_Clear(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Clear(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#Rebuild. static PyObject* index_Rebuild(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Rebuild(); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#Synchronize. static PyObject* index_Synchronize(PyIndex* self, PyObject* pyargs) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pyhard = PyTuple_GET_ITEM(pyargs, 0); const bool hard = PyObject_IsTrue(pyhard); tkrzw::Status status(tkrzw::Status::SUCCESS); { NativeLock lock(self->concurrent); status = self->index->Synchronize(hard); } return CreatePyTkStatusMove(std::move(status)); } // Implementation of Index#IsOpen. static PyObject* index_IsOpen(PyIndex* self) { if (self->index == nullptr) { Py_RETURN_FALSE; } Py_RETURN_TRUE; } // Implementation of Index#IsWritable. static PyObject* index_IsWritable(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } const bool writable = self->index->IsWritable(); if (writable) { Py_RETURN_TRUE; } Py_RETURN_FALSE; } // Implementation of Index#MakeIterator. static PyObject* index_MakeIterator(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } PyTypeObject* pyitertype = (PyTypeObject*)cls_indexiter; PyIndexIterator* pyiter = (PyIndexIterator*)pyitertype->tp_alloc(pyitertype, 0); if (!pyiter) return nullptr; { NativeLock lock(self->concurrent); pyiter->iter = self->index->MakeIterator().release(); } pyiter->concurrent = self->concurrent; return (PyObject*)pyiter; } // Implementation of Index#__len__. static Py_ssize_t index_len(PyIndex* self) { if (self->index == nullptr) { return 0; } int64_t count = -1; { NativeLock lock(self->concurrent); count = self->index->Count(); } return std::max(count, 0); } // Implementation of Index#__contains__. static int index_contains(PyIndex* self, PyObject* pyrec) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return -1; } if (!PySequence_Check(pyrec)) { ThrowInvalidArguments("not sequence argument"); return -1; } if (PySequence_Size(pyrec) != 2) { ThrowInvalidArguments("not pair argument"); return -1; } PyObject* pykey = PySequence_GetItem(pyrec, 0); PyObject* pyvalue = PySequence_GetItem(pyrec, 1); SoftString key(pykey); SoftString value(pyvalue); bool check = false; { NativeLock lock(self->concurrent); check = self->index->Check(key.Get(), value.Get()); } Py_DECREF(pykey); Py_DECREF(pyvalue); return check ? 1 : 0; } // Implementation of Index#__iter__. static PyObject* index_iter(PyIndex* self) { if (self->index == nullptr) { ThrowInvalidArguments("not opened index"); return nullptr; } PyTypeObject* pyitertype = (PyTypeObject*)cls_indexiter; PyIndexIterator* pyiter = (PyIndexIterator*)pyitertype->tp_alloc(pyitertype, 0); if (!pyiter) return nullptr; { NativeLock lock(self->concurrent); pyiter->iter = self->index->MakeIterator().release(); pyiter->concurrent = self->concurrent; pyiter->iter->First(); } return (PyObject*)pyiter; } // Defines the Index class. static bool DefineIndex() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.Index"; pytype.tp_basicsize = sizeof(PyIndex); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Secondary index."; pytype.tp_new = index_new; pytype.tp_dealloc = (destructor)index_dealloc; pytype.tp_init = (initproc)index_init; pytype.tp_repr = (unaryfunc)index_repr; pytype.tp_str = (unaryfunc)index_str; static PyMethodDef methods[] = { {"Open", (PyCFunction)index_Open, METH_VARARGS | METH_KEYWORDS, "Opens an index file."}, {"Close", (PyCFunction)index_Close, METH_NOARGS, "Closes the index file."}, {"GetValues", (PyCFunction)index_GetValues, METH_VARARGS, "Gets all values of records of a key."}, {"GetValuesStr", (PyCFunction)index_GetValuesStr, METH_VARARGS, "Gets all values of records of a key, as strings."}, {"Add", (PyCFunction)index_Add, METH_VARARGS, "Adds a record."}, {"Remove", (PyCFunction)index_Remove, METH_VARARGS, "Removes a record."}, {"Count", (PyCFunction)index_Count, METH_NOARGS, "Gets the number of records."}, {"GetFilePath", (PyCFunction)index_GetFilePath, METH_NOARGS, "Gets the path of the index file."}, {"Clear", (PyCFunction)index_Clear, METH_NOARGS, "Removes all records."}, {"Rebuild", (PyCFunction)index_Rebuild, METH_NOARGS, "Rebuilds the entire index."}, {"Synchronize", (PyCFunction)index_Synchronize, METH_VARARGS, "Synchronizes the content of the index to the file system."}, {"IsOpen", (PyCFunction)index_IsOpen, METH_NOARGS, "Checks whether the index is open."}, {"IsWritable", (PyCFunction)index_IsWritable, METH_NOARGS, "Checks whether the index is writable."}, {"MakeIterator", (PyCFunction)index_MakeIterator, METH_NOARGS, "Makes an iterator for each record."}, {nullptr, nullptr, 0, nullptr}, }; pytype.tp_methods = methods; static PyMappingMethods map_methods; std::memset(&map_methods, 0, sizeof(map_methods)); map_methods.mp_length = (lenfunc)index_len; pytype.tp_as_mapping = &map_methods; static PySequenceMethods seq_methods; std::memset(&seq_methods, 0, sizeof(seq_methods)); seq_methods.sq_contains = (objobjproc)index_contains; pytype.tp_as_sequence = &seq_methods; pytype.tp_iter = (getiterfunc)index_iter; if (PyType_Ready(&pytype) != 0) return false; cls_index = (PyObject*)&pytype; Py_INCREF(cls_index); if (PyModule_AddObject(mod_tkrzw, "Index", cls_index) != 0) return false; return true; } // Implementation of IndexIterator.new. static PyObject* indexiter_new(PyTypeObject* pytype, PyObject* pyargs, PyObject* pykwds) { PyIndexIterator* self = (PyIndexIterator*)pytype->tp_alloc(pytype, 0); if (!self) return nullptr; self->iter = nullptr; self->concurrent = false; return (PyObject*)self; } // Implementation of IndexIterator#dealloc. static void indexiter_dealloc(PyIndexIterator* self) { delete self->iter; Py_TYPE(self)->tp_free((PyObject*)self); } // Implementation of IndexIterator#__init__. static int indexiter_init(PyIndexIterator* self, PyObject* pyargs, PyObject* pykwds) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc != 1) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return -1; } PyObject* pyindex_obj = PyTuple_GET_ITEM(pyargs, 0); if (!PyObject_IsInstance(pyindex_obj, cls_index)) { ThrowInvalidArguments("the argument is not an Index"); return -1; } PyIndex* pyindex = (PyIndex*)pyindex_obj; { NativeLock lock(pyindex->concurrent); self->iter = pyindex->index->MakeIterator().release(); } self->concurrent = pyindex->concurrent; return 0; } // Implementation of IndexIterator#__repr__. static PyObject* indexiter_repr(PyIndexIterator* self) { std::string key; { NativeLock lock(self->concurrent); if (!self->iter->Get(&key)) { key = "(unlocated)"; } } return CreatePyString(tkrzw::StrCat( "")); } // Implementation of IndexIterator#__str__. static PyObject* indexiter_str(PyIndexIterator* self) { std::string key; { NativeLock lock(self->concurrent); if (!self->iter->Get(&key)) { key = "(unlocated)"; } } return CreatePyString(tkrzw::StrEscapeC(key, true)); } // Implementation of IndexIterator#First. static PyObject* indexiter_First(PyIndexIterator* self) { { NativeLock lock(self->concurrent); self->iter->First(); } Py_RETURN_NONE; } // Implementation of IndexIterator#Last. static PyObject* indexiter_Last(PyIndexIterator* self) { { NativeLock lock(self->concurrent); self->iter->Last(); } Py_RETURN_NONE; } // Implementation of IndexIterator#Jump. static PyObject* indexiter_Jump(PyIndexIterator* self, PyObject* pyargs) { const int32_t argc = PyTuple_GET_SIZE(pyargs); if (argc < 1 || argc > 2) { ThrowInvalidArguments(argc < 1 ? "too few arguments" : "too many arguments"); return nullptr; } PyObject* pykey = PyTuple_GET_ITEM(pyargs, 0); SoftString key(pykey); if (argc > 1) { PyObject* pyvalue = PyTuple_GET_ITEM(pyargs, 1); SoftString value(pyvalue); NativeLock lock(self->concurrent); self->iter->Jump(key.Get(), value.Get()); } else { NativeLock lock(self->concurrent); self->iter->Jump(key.Get()); } Py_RETURN_NONE; } // Implementation of IndexIterator#Next. static PyObject* indexiter_Next(PyIndexIterator* self) { { NativeLock lock(self->concurrent); self->iter->Next(); } Py_RETURN_NONE; } // Implementation of IndexIterator#Previous. static PyObject* indexiter_Previous(PyIndexIterator* self) { { NativeLock lock(self->concurrent); self->iter->Previous(); } Py_RETURN_NONE; } // Implementation of IndexIterator#Get. static PyObject* indexiter_Get(PyIndexIterator* self) { std::string key, value; bool ok = false; { NativeLock lock(self->concurrent); ok = self->iter->Get(&key, &value); } if (ok) { PyObject* pykey = CreatePyBytes(key); PyObject* pyvalue = CreatePyBytes(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of IndexIterator#GetStr. static PyObject* indexiter_GetStr(PyIndexIterator* self) { std::string key, value; bool ok = false; { NativeLock lock(self->concurrent); ok = self->iter->Get(&key, &value); } if (ok) { PyObject* pykey = CreatePyString(key); PyObject* pyvalue = CreatePyString(value); PyObject * pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pyvalue); Py_DECREF(pykey); return pyrv; } Py_RETURN_NONE; } // Implementation of IndexIterator#__next__. static PyObject* indexiter_iternext(PyIndexIterator* self) { std::string key, value; bool ok = false; { NativeLock lock(self->concurrent); ok = self->iter->Get(&key, &value); } PyObject* pyrv = nullptr; if (ok) { PyObject* pykey = CreatePyBytes(key); PyObject* pyvalue = CreatePyBytes(value); pyrv = PyTuple_Pack(2, pykey, pyvalue); Py_DECREF(pykey); Py_DECREF(pyvalue); self->iter->Next(); } else { PyErr_SetString(PyExc_StopIteration, "end of iteration"); pyrv = nullptr; } return pyrv; } // Defines the IndexIterator class. static bool DefineIndexIterator() { static PyTypeObject pytype = {PyVarObject_HEAD_INIT(nullptr, 0)}; const size_t zoff = offsetof(PyTypeObject, tp_name); std::memset((char*)&pytype + zoff, 0, sizeof(pytype) - zoff); pytype.tp_name = "tkrzw.IndexIterator"; pytype.tp_basicsize = sizeof(PyIndexIterator); pytype.tp_itemsize = 0; pytype.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; pytype.tp_doc = "Iterator for each record of the secondary index."; pytype.tp_new = indexiter_new; pytype.tp_dealloc = (destructor)indexiter_dealloc; pytype.tp_init = (initproc)indexiter_init; pytype.tp_repr = (unaryfunc)indexiter_repr; pytype.tp_str = (unaryfunc)indexiter_str; static PyMethodDef methods[] = { {"First", (PyCFunction)indexiter_First, METH_NOARGS, "Initializes the iterator to indicate the first record."}, {"Last", (PyCFunction)indexiter_Last, METH_NOARGS, "Initializes the iterator to indicate the last record."}, {"Jump", (PyCFunction)indexiter_Jump, METH_VARARGS, "Initializes the iterator to indicate a specific range."}, {"Next", (PyCFunction)indexiter_Next, METH_NOARGS, "Moves the iterator to the next record."}, {"Previous", (PyCFunction)indexiter_Previous, METH_NOARGS, "Moves the iterator to the previous record."}, {"Get", (PyCFunction)indexiter_Get, METH_NOARGS, "Gets the key and the value of the current record of the iterator."}, {"GetStr", (PyCFunction)indexiter_GetStr, METH_NOARGS, "Gets the key and the value of the current record of the iterator, as strings."}, {nullptr, nullptr, 0, nullptr} }; pytype.tp_methods = methods; pytype.tp_iternext = (iternextfunc)indexiter_iternext; if (PyType_Ready(&pytype) != 0) return false; cls_indexiter = (PyObject*)&pytype; Py_INCREF(cls_indexiter); if (PyModule_AddObject(mod_tkrzw, "IndexIterator", cls_indexiter) != 0) return false; return true; } // Entry point of the library. PyMODINIT_FUNC PyInit_tkrzw() { if (!DefineModule()) return nullptr; if (!DefineUtility()) return nullptr; if (!DefineStatus()) return nullptr; if (!DefineStatusException()) return nullptr; if (!DefineFuture()) return nullptr; if (!DefineDBM()) return nullptr; if (!DefineIterator()) return nullptr; if (!DefineAsyncDBM()) return nullptr; if (!DefineFile()) return nullptr; if (!DefineIndex()) return nullptr; if (!DefineIndexIterator()) return nullptr; return mod_tkrzw; } } // extern "C" // END OF FILE tkrzw-python-0.1.32/perf.py0000775000175000017500000001502214626642264014626 0ustar mikiomikio#! /usr/bin/python3 # -*- coding: utf-8 -*- #-------------------------------------------------------------------------------------------------- # Performance tests # # Copyright 2020 Google LLC # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file # except in compliance with the License. You may obtain a copy of the License at # https://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the # License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, # either express or implied. See the License for the specific language governing permissions # and limitations under the License. #-------------------------------------------------------------------------------------------------- import argparse import asyncio import os import re import random import shutil import sys import threading import time from tkrzw import * # main routine def main(argv): ap = argparse.ArgumentParser( prog="perf.py", description="Performance Checker", formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--path", default="") ap.add_argument("--params", default="") ap.add_argument("--iter", type=int, default=10000) ap.add_argument("--threads", type=int, default=1) ap.add_argument("--async_threads", type=int, default=0) ap.add_argument("--random", action='store_true', default=False) args = ap.parse_args(argv) path = args.path open_params = {} open_params_exprs = [] for field in args.params.split(","): columns = field.split("=", 1) if len(columns) == 2: open_params[columns[0]] = columns[1] open_params_exprs.append(columns[0] + "=" + columns[1]) num_iterations = args.iter num_threads = args.threads is_random = args.random print("path: {}".format(path)) print("params: {}".format(",".join(open_params_exprs))) print("num_iterations: {}".format(num_iterations)) print("num_threads: {}".format(num_threads)) print("is_random: {}".format(is_random)) print("") open_params["truncate"] = True start_mem_usage = Utility.GetMemoryUsage() dbm = DBM() dbm.Open(path, True, **open_params).OrDie() adbm = None if args.async_threads > 0: adbm = AsyncDBM(dbm, args.async_threads) class Setter(threading.Thread): def __init__(self, thid): threading.Thread.__init__(self) self.thid = thid def run(self): rnd_state = random.Random(self.thid) for i in range(0, num_iterations): if is_random: key_num = rnd_state.randint(1, num_iterations * num_threads) else: key_num = self.thid * num_iterations + i key = "{:08d}".format(key_num) if adbm: adbm.Set(key, key).Get().OrDie() else: dbm.Set(key, key).OrDie() seq = i + 1 if self.thid == 0 and seq % (num_iterations / 500) == 0: print(".", end="") if seq % (num_iterations / 10) == 0: print(" ({:08d})".format(seq)) sys.stdout.flush() print("Setting:") start_time = time.time() threads = [] for thid in range(0, num_threads): th = Setter(thid) th.start() threads.append(th) for th in threads: th.join() dbm.Synchronize(False).OrDie() end_time = time.time() elapsed = end_time - start_time mem_usage = Utility.GetMemoryUsage() - start_mem_usage print("Setting done: num_records={:d} file_size={:d} time={:.3f} qps={:.0f} mem={:d}".format( dbm.Count(), dbm.GetFileSize() or -1, elapsed, num_iterations * num_threads / elapsed, mem_usage)) print("") class Getter(threading.Thread): def __init__(self, thid): threading.Thread.__init__(self) self.thid = thid def run(self): rnd_state = random.Random(self.thid) for i in range(0, num_iterations): if is_random: key_num = rnd_state.randint(1, num_iterations * num_threads) else: key_num = self.thid * num_iterations + i key = "{:08d}".format(key_num) if adbm: status, value = adbm.Get(key).Get() else: status = Status() value = dbm.Get(key, status) if status != Status.SUCCESS and status != Status.NOT_FOUND_ERROR: raise RuntimeError("Get failed: " + str(status)) seq = i + 1 if self.thid == 0 and seq % (num_iterations / 500) == 0: print(".", end="") if seq % (num_iterations / 10) == 0: print(" ({:08d})".format(seq)) sys.stdout.flush() print("Getting:") start_time = time.time() threads = [] for thid in range(0, num_threads): th = Getter(thid) th.start() threads.append(th) for th in threads: th.join() end_time = time.time() elapsed = end_time - start_time mem_usage = Utility.GetMemoryUsage() - start_mem_usage print("Getting done: num_records={:d} file_size={:d} time={:.3f} qps={:.0f} mem={:d}".format( dbm.Count(), dbm.GetFileSize() or -1, elapsed, num_iterations * num_threads / elapsed, mem_usage)) print("") class Remover(threading.Thread): def __init__(self, thid): threading.Thread.__init__(self) self.thid = thid def run(self): rnd_state = random.Random(self.thid) for i in range(0, num_iterations): if is_random: key_num = rnd_state.randint(1, num_iterations * num_threads) else: key_num = self.thid * num_iterations + i key = "{:08d}".format(key_num) if adbm: status = adbm.Remove(key).Get() else: status = dbm.Remove(key) if status != Status.SUCCESS and status != Status.NOT_FOUND_ERROR: raise RuntimeError("Remove failed: " + str(status)) seq = i + 1 if self.thid == 0 and seq % (num_iterations / 500) == 0: print(".", end="") if seq % (num_iterations / 10) == 0: print(" ({:08d})".format(seq)) sys.stdout.flush() print("Removing:") start_time = time.time() threads = [] for thid in range(0, num_threads): th = Remover(thid) th.start() threads.append(th) for th in threads: th.join() dbm.Synchronize(False).OrDie() end_time = time.time() elapsed = end_time - start_time mem_usage = Utility.GetMemoryUsage() - start_mem_usage print("Removing done: num_records={:d} file_size={:d} time={:.3f} qps={:.0f} mem={:d}".format( dbm.Count(), dbm.GetFileSize() or -1, elapsed, num_iterations * num_threads / elapsed, mem_usage)) print("") dbm.Close().OrDie() return 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:])) # END OF FILE